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ABSTRACT 

This thesis examines the problems affecting the 
guantitative assessment of technical risk in Department of 
Defense major weapon systems acquisition. A Decision Theory 
approach is geea: Commercial techniques and current DoD 
methods of technical risk assessment are investigated. 
TASCFORM™™ technology values are used in a linear regression 
model to characterize the growth of technology over time. 
The model residuals provide a probability distribution for 
estimating the likelihood of achieving a specified level of 
technical performance. The benefit of a utility function 
for describing technical risk perceptions feck oneiaesda:. 
The Expanded Pearson-Tukey method of describing risk is also 
investigated. Continued research into technology valuation 
techniques is recommended. A test case application of the 
Expanded Pearson-Tukey method is also recommended, to. 


determine its ability to provide reliable and timely 


quantitative technical risk information. 
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I. INTRODUCTION 
A. BACKGROUND 
Soviet military retrenchment and political collapse have 
Significantly affected U.S. strategic and tactical force 
evriceuns and requirements. United States national defense 
goals and requirements presently lack the refinement and 


broad consensus forty years of Soviet confrontation 





engendered. The Federal budget deficit is viewed as a 
severe threat to the health and future of the nation's 
economy and the Department of Defense (DoD) budget will 
remain under severe pressure throughout the 1990s. Not only 
will there be less money to support existing programs and 
initiatives, but the Department's ability to introduce new 
systems will be increasingly constrained. 

The systems purchased to support the changes in military 
doctrine that an altered international landscape requires 
must have greater flexibility, reliability and 
interoperability. Simultaneously, these systems must be 
brought on-line without the schedule and cost over-rung that 
plagued DoD during the years of sustained Soviet threat. 
Acquisition programs that. experience perturbations in terms 
of schedule, cost, or performance will not only fail to 


mature but will cause damaging ripple effects to other 


programs because of the smaller total force structure. 








For the last forty years the United States has relied 
upon an acquisition strategy of fielding technologically 
Superior systems to counter—balance the numerical advantages 
of a perceived Soviet threat. Technical sophistication has * 
permeated all levels of force structure and doctrine, 
‘requiring state-of-the-art performance from individual 
hardware eonponenee: Increasingly complex data acquisition, 
processing, and exchange networks are required to support 
the command and control requirements of forces equipped with 
these systems. | 

Despite continuous efforts, the United States has been 
unable to consistently develop, field, and maintain a force 
structure that possesses an effective technological 
advantage over Warsaw Pact/Soviet forces [Ref. 1]. This has 
forced the U.S. defense establishment to continually 
attempt the development and fielding of ever more advanced 
capabilities subject to increasingly rigid cost and time 
constraints. Aggressive performance and schedule goals 
sometimes have been specified without a consistent or 
coherent assessment of the risks. 

While much criticism has been leveled at DoD because of 
past cost and schedule over runs and performance failures 
[Ref. 2:pp. xxiii-xxv], there was often sufficient breadth 

in new capability development efforts and elasticity in 


budgets to ameliorate the effects of performance : 


short-falls, cost, and schedule over-runs in any one 





particular program. The Navy was able to field improvements > 
to Ant ini¥ Warfare (AAW) capabilities with three systems 
through the 1970s and 1980s. The AEGIS ship-board radar and 
missile system, F-14 Tomcat/PHOENIX missile system, and the 
F/A-18 Hornet, were eventually fielded despite cost, 
schedule, and performance problems with all three. In the 
future, programs which experience these problems could have 
a devastating impact on force structures and capabilities. 

| The Navy's A-12 Avenger program, employing "stealth" 
technology, was designed as a replacement for the venerable 
A-6 Intruder medium attack aircraft. The A-12 enjoyed the 
highest priority within the service. An A-6 Upgrade program 
(A-6F) was cancelled so that resources available for the A- 
12 program would not be constrained. Cancellation of the A- 
12 program not only resulted in the significant delay of a 
vitally needed capability, but has seriously weakened the 
Navy's ability to fund other important aviation programs in 
a climate of fiscal austerity. Financial resources 66 meet 
heavy lift, maritime patrol, AAW, and carrier-based early 
warning programs are all jeopardized [Refs. 3, 4]. 

The current acquisition environment is characterized by 
concurrent deve lopment of systems that will operate with a 
wide variety of forces in numerous environments. Single- 
service programs tailored to meet unique service objectives 


will receive far greater scrutiny and will become rarities. 


The impact of a termination could have severe effects on the 











capabilities of the entire DoD force structure in this sort 
of environment. Termination of the A-12 has stopped Air 
Force efforts to develop a derivative A-12 as a replacement 


for the F-15 Eagle and F-111 [Ref. 3]. | . 


B. OBJECTIVE 


The objective of this thesis is to examine existing 
acquisition program risk assessment theories and methods, 
evaluate their efficacy as employed in commercial settings, 
and look for ways that successful commercial approaches. 
could be adapted for use within the unique DoD acquisition 
decision environment. Specifically, this reseeren attempts 
to wees a useful and flexible definition of technological 
risk and to identify risk assessment methods that are 


accessible to DoD program managers and their staffs. 


C. RESEARCH QUESTIONS 

Commencing with the axiomatic framework constructed by 
von Neumann and Morgenstern [Ref. 5], a large body of work 
has addressed risk assessment for individual or corporate 
decisions. Transference of these theories and techniques to 
the DoD acquisition setting requires investigation. 
Assumptions about the nature and availability of 
alternatives for subsequent "risk premium” and certainty 
equivalent analysis may not directly apply to the DoD 


environment. This thesis will examine the following aspects 


of risk assessment from a Decision Theory perspective: 








1. What are the underlying theories and methodology 
supporting risk assessment in decision analysis? 

2. How are these methods applied in commercial 
settings? What are their strengths and weaknesses? 

3. What are the major factors defining the DoD 
technical risk assessment environment? Are these factors 
Similar to those found in the commercial world? 

4. Evaluate whether or not the techniques used by 
commercial acquisition program managers can be applied 
directly to the DoD acquisition case. If not, what changes 
our be required. 

5. Suggest possible improvements to existing DoD 
technical risk acquisition methods. | 

This thesis will not examine cost and/or schedule issues 
nor the procedures for determining the performance 


requirements specified for new systems. 


D. SCOPE AND LIMITATIONS 
1. Scope 
There are two basic approaches used to define risk 
assessment: normative or prescriptive, and descriptive. The 
normative approach strives to define ways in which a 
"rational" person should go about confronting the Universe. 


The descriptive approach primarily seeks to understand how 


real people make real decisions. 





There exists a resultant tension between these two 
approaches that must be monitored if usable decision tools 
are to be fashioned. However, for the purposes of this 
iene: the emphasis will be on how a normative approach can 
be applied to the risk assessment problem of the program 
manager or acquisition executive. 

2. Limitations 

Within the normative scheme there are three 
dec isienemacing approaches, all using some form of “divide 
and conquer": 

1. Cost Benefit Theory (CBT) 

2. Social Welfare Theory (SWT) 

3. Decision Theory (DT) 
All three use a standard partitioning of the decision-making 
problem, seeking to counter-—pose knowledge and values at 
each step of the process and synthesize these in a logically 
consistent manner [Ref. 6]. 

The CBT approach is basically an impersonal one, 
based upon "scientific objectivity.” SWT tends to focus on 
the organizational dynamics and social processes. Decision 
Theory frames the problem within the context of the 
individual as the decision-making entity. 

While subjective in its world view, Decision Theory 


seems suitable for helping individual acquisition decision 


makers confronted with the problem of technical risk 





% 
“« 


assessment. This is particularly beneficial, since a body 





of objective, verifiable data is not always available at the 


-early stages of new acquisition projects [Ref. 7]. 


E. LITERATURE REVIEW 

One paves cules aspect of the DoD acquisition conundrum 
is the assessment of technical risk. A 1986 GAO Report, 
commissioned by the U.S. Senate Committee on Governmental 
Affairs, found numerous inconsistencies and omissions in how 


acquisition program offices tackled the problem of technical 


risk assessment. The report examined twenty-five major 


acquisition programs whose development and production costs 


exceeded $180 billion: 


DOD has identified many technical risk approaches, both 
quantitative and qualitative. But there is insufficient 
policy and training to guide program managers in the 
selection of suitable approaches. Further, no standard 
definition of technical risk exists within DOD. 
Accordingly, many program offices have developed their 
own informal definitions of technical risk and risk- 
rating categories, but GAO found them inconsistent and 
sometimes contradictory. Despite DOD's 1981 initiative, 
none of the 25 program offices had conducted a 
quantitative technical risk assessment to support 
budgeting for risk. [Ref. 8:p. 3] 


To date, the problems enumerated by the GAO in 1986 
remain largely unaddressed. While a large number of 
logically consistent decision tools have been developed in 
academic environments, accessibility and implementation 


by DoD acquisition authorities and their staffs remain 


inconsistent or even nonexistent. 





Despite the daunting challenges involved in developing 
such a revolutionary stealth capability. the Navy 
characterized the A-12 program as possessing "low" risk. 

The Navy designated the T-45 program, a jet training 
aircraft to replace both the T-2 Buckeye and TA-4 Skyhawk 
training aircraft, also as "“low" risk [Ref. 9]. 

The T-45 was originally a land-based British design that 
had to be altered for compatibility with the carrier 
environment. The "low-risk" designation was assigned, 
despite the fact that no land-based aircraft design had been 
successfully reengineered for carrier duty since 1945. The 
program suffered serious performance problems, schedule 
slippages, and cost overruns [Ref. 9]. 

The problem of aircraft acquisition technical risk 
assessment is not confined to aerodynamic issues. The 
Navy's Enhanced Modular Signal Processor (EMSP), designed to 
handle a wide range of acoustic and electronic processing 
requirements through the end of the twentieth century, has 
had by many problems despite its designation as a "low" risk 
program [Ref. 10]. The Air Force' C-17 program has been 
beset by performance and cost problems as well as schedule 
delays arising from the adoption of an ostensibly "low risk" 


approach to the software design for flight control computers 


[Ref. 11). 





Despite numerous DoD sna serv teeaeaued directives aimed | 
at assessing, quantifying and controlling risk, e.g-., DoD 
Instruction 5000.2, there are no DoD mandated methodologies 
for meeting these directives [Refs. 12, 13, 14]. The 
Defense Systems Management College (DSMC) has issued a 


publication that covers most of the standard methods for 





risk identification and assessment [Ref. 15]. However, 
there has been no adoption of either the DSMC's scale for 
low, medium, or high probability of occurrence, nor eae 
policy guidance concerning what should constitute low, 


medium, or high risk for acquisition activities. 


F. ORGANIZATION AND CHAPTER SUMMARY 

This chapter gave on overview of the importance and 
problems associated with ie assessment of technical risk in 
DoD acquisition programs. The objective of the thesis was 
presented. along with limitations to problem definition and 
subsequent research. The aim of the work will be to examine 
how the tenets of Decision Theory can be applied to the 
problem of DoD acquisition program technical risk aoe saneak 
in a manner that is accessible to DoD acquisition decision-— 
makers. 

chapter II will present specific background information 
and issues necessary for understanding the theoretical basis 


of Decision Theory and its limitations. Uncertainty and 


risk perception interactions will also be discussed. 





Chapter III will look at how technical risk assessment 


- igsues are addressed in commercial operations. Chapter IV 





will define the DoD technical risk assessment and decision 
environment, a necessity for framing any recommendations for 
potential pise assessment approaches. Chapter V will 
attempt to provide some recommended methods for technical 
pie assessment. Chapter VI will contain concluding 
remarks, a summary of major findings and recommendations for 


future research. 
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II. BACKGROUND AND THEORETICAL FRAMEWORK 


A. THE DECISION PROBLEM 
Decision making iS a unique human activity. Only humans 

seem to possess the faculty of recognizing the possibility 
or availability of more than one alternative or course of 
action in a set of circumstances over some period of time. 
Decision making arises from the application of some value 
Weseen imposed upon a particular environment. It assumes 
that a selected objective or goal can be obtained through 


the purposeful expenditure of resources in hand, ". . . in 





the light of norms or general principles, or of future 
outcomes." (Ref. 16:p. 131] | 

As Bunn points out, the activity of decision making is 
available to individuals, organizations and society. 
Decision making is a characteristic of pounposeni systems: 
the nature and gravity of decision HAAG activities _ 
tied to the attainment of objects or aes that in some 
fashion support or further the "good" of the entity 
formulating the decision [Ref. 17]. Simon's definition of 
the three phases of decision making applies to the DoD 
acquisition environment: Finding occasions for making 
decisions; finding possible courses of action; and choosing 


among courses of action." [Ref. 18]. 


11 





The purpose of decision analysis is to provide a 
logically consistent and flexible framework for helping the 
decision maker generate and evaluate the various 
alternatives that may present tenaeiwes.. The flaws 
inherent in innate and intuitive decision making processes 
have been the subject of extensive research. Numerous 
researchers have commented upon and measured the relatively 
limited human data processing capabilities [Ref. 19]. 
Decision types fall into three categories. There ave 
automatic or intuitive decisions that take place in such a 
iene that the individual is often unaware of the process. 
Decision making in an athletic competition is an example of 
this sort of decision making. Pondering the proper course 
of action during an athletic event almost invariably seems 
to impair performance. 

Some decisions may be very complex but exhibit 
characteristics that make their resolution tractable to 
strict rules or instructions. On a mechanical level, the 
operation of a thermostat or a fuel control unit falls into 
this category. The operator may not know instinctively what 
action to take. However, a satisfactory result can be 
obtained from the careful and sequential following of a set 
of instructions, achieving a desired temperature or power 
setting without weighing options or detailed process 


knowledge. 


Ve 
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Lastly, there are those decisions that require careful 
and measured consideration. These are often characterized 
by a bewildering array of possible alternatives. Obtaining 
the knowledge necessary to determine the suitability and 
cost of achieving a particular course of action may be 
extraordinarily difficult. Howard states, "decision making 
is what you do when you do not know what to do." (Ref. 20]. 


1. Elements of the Decision Problem 





Following Bunn [Ref. 17], four major factors 
characterize the problem and affect the subsequent 
formulation of any solution: Uncertainty, Multiple 
Objectives, Multiple Options, and Sequentiality. 

| a. Uncertainty 

Uncertainty is the Key obstacle to the 
resolution of most important decisions and affects the other 
three factors. Many times the decision maker can envision 
multiple possible outcomes and can place some sort of odds 
of occurrence on particular events. Some of the outcomes 
will be counter to the goals pursued and might be ruinous to 
the individual or organization making the decision. 
Decision makers simply cannot know, a priori, how events 
will proceed once a particular course of action is set into 


motion. 
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b. Multiple Objectives 
Organizational or individual goals may be 
complex when carefully considered. The objective may have 
multiple attributes. A computer system could be 
characterized by the length of words it can handle, 
processing time, or memory size. The goal may be subject to 
some resource constraint or other conflicting requirement. 
The problems associated with nuclear power plant design and 
siting (Ref: 21] are classic examples of this dilemma. How 
should the decision maker evaluate independent and possibly 
conflicting goals and attributes? 
c. Multiple Options 
The alternatives that are generated as 
candidates for meeting a goal may not completely satisfy all 
requirements. Some combination of alternatives may be 
required to reach the Sbiective. How should the decision 
maker eliminate inadequate options without unnecessarily 
reducing the resolution of any screening device? How can he 
ensure that he has not overlooked an efficient possibility? 
dad. Sequentiality | 
Problems often present themselves over a period 
of time: the entire issue is not resolvable at any 
particular instant but requires continual iteration as new 
data become available. The decision maker may be confronted 
with numerous paths whose existence and efficiency are 


contingent upon previous actions. Sequentiality may be 
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deliberately imposed upon the problem by the organization. 
In his analysis of the Cuban Missile Crisis, Allison 
{Ref. 22] discusses senior decision makers' penchants for 
delaying the implementation of decisions that may be 
irreversible or delaying action because of fear that the set 
of possible options may be reduced. 
2. Decision Analysis Requirements 
| The purpose of decision analysis is to directly 

address issues in amanner that helps the decision maker 
more completely define his problem and evaluate his possible 
courses of action. The analyst must remember that someone 
else will make the decision and will carry the onus of the 
pesuite ap events turn out poorly. But what are the , 
requirements for a good decision analysis? First, the 
analysis must capture the essential elements of the problem 
in a fashion that the decision maker can grasp. This 
pieuunne see an interative format, relying upon the intuition 
and judgement of those charged with arriving at a decision. 
The investigation must be couched in such a manner that it 
leads to a complete understanding of the issue at hand, or 
at least as complete as is possible, given the ever-present 
constraints of time and money [Ref. 23]. 

Second, the rule of coherence must be followed. If 
the decision maker Pecks to behave ina rational manner, his 
actions must be logically consistent and free from hidden 


contradictions. Rationality as a necessary condition in 
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this sense implies that the individual or other decision 
making entity must seek to maintain or improve his wealth. 
Failure to act in a coherent fashion leads to the "perpetual 
money-making machine" paradox. 

Suppose an individual considered event Aas less 
likely than event B and event B as less likely than event C. 
Suppose further that rather than concluding that A is less 
likely than C, this individual decides that C is less likely 
than A. Assume that event A provides an outcome that is 
valuable to this individual, a prize of some sort, while B 
and C do not. In this case, the person would be willing to 
pay some amount of money to replace A with B. If the prize 
is now contingent on B, the individual could be induced to 
pay some fee to replace B with C. A third sum of money 
could be obtained by offering to replace C with A, an event 
he considers less likely. The individual is now at the 
starting point with the exception that he is demonstrably 
poorer: As long as the individual holds to his order of 
ranking, money can be made ad infinitum, or at least until 
he is drained of cash. Such behavior is defined as 


incoherent [Ref. 17]. 


B. DECISION CRITERIA 
Decision criteria can be separated into two main 
categories, nonstochastic and stochastic. The nonstochastic | . 


approach ignores the existence of probability and seeks to 
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use a restrictive concept of dominance to generate the best 





solution. Stochastic techniques explicitly apply notions of 
probability to the process of determining a best solution. 
There are problems associated with the application of both 
these approaches. 3 
1. Nonstochastic Methods 

-Nonstochastic methods seek to discover the best 
eieeyaeace for satisfying the decision maker's goals 
through the discovery of a course of action that will be 
superior to all others, no matter the outcome of events. 
Dominance approaches can be defined as strict dominance, 
maximin, maximax, and regret strategies. 

a. Outcome Dominance 

Consider a matrix that lists possible results 

for three different alternatives in the event of three 
different outcomes and assume that the aoe iaisn maker 


prefers more to less: 


ALTERNATIVES 


Ei :2 it e 
POSSIBLE E2 16 2 4 (1) 
OUTCOMES 

Ex 19 7 8 
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An inspection shows that for any possible event, 
Ei, Ez, or Ex, alternative A: dominates Az. As summarized 
by Bunn, any alternative Ay. dominates A,, if for every Ei 
oe is greater than or equal to yip, and yig is greater than 
Yip for at least one E: [Ref. 17:p. 17]. No assumptions 
about the relative likelihood of event occurrence have been 
made. While this technique is simple to use, it is only. 
applicable to a very small set of fortunate circumstances. 
Additionally, the best this technique can be expected to 
provide is some reduction in the number of possibly 
satisfactory alternatives. As can be seen from (1) above, 
while alternative Az can be removed from consideration, 
deciding between A, and A» remains a problem. 

b. Maximin 

The maximin approach takes a pessimistic or 
conservative approach to the problem to the sorting of best 
alternatives. Consider (1) again. The maximin approach is 
to search each column and identify the smallest outcome 
value a particular event could generate. For this example, 


the column minimums have been identified by an asterisk. 
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ALTERNATIVES 
1Aa Az A-x 
Ei tae 1 3¥ 


POSSIBLE Ezz 


‘6 5 4 ay. 
‘OUTCOMES . | 


The maximin solution problem is to choose the alternative 
that provides the largest of the possible minimum, in this 
case Ax. Maximin attempts to make the best of a bad 
Situation. 

The efficacy of this approach diminishes rapidly 
in cases where there are very large differences in possible 
payoffs. Bunn [Ref. 17:p. 18] provides the following 


counter-example: 





ALTERNATIVES 
iA, Ae 
POSSIBLE Ei! 31 32 
OUTCOMES 7 _— (3) 
Ez! 10,000 33 


Here the maximin strategy would recommend Az based upon a 
strict observance of the decision rule. The decision maker 
following this recommendation, assuming again that the 
object is maximize monetary returns, would end up “penny- 


wise and pound-foolish." 
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c.  Maximax 
This strategy is the obverse of maximin, viewing 

the occurrence of random events in the most optimistic 
manner. The decision maker is required to identify the 
maximum payoff for each possible course of action and then 
select the alternative that provides the greatest return. 
This is clearly untenable in the event that an alternative 
could cause a very large loss. Consider the following 


payoff matrix: 


ALTERNATIVES 
© Ag Az 
POSSIBLE ot , 
- OUTCOMES Ei: Sl 32 
(4) 
Ee: —10,000 33 


Maximax would recommend alternative Az, oblivious to the 
potential for suhstantial monetary loss. 
da. Regret 
The regret strategy uses an approach 
Seaiiey to maximin. The sevedtion criterion focuses upon 
how the decision might be viewed from the vantage of 
Haid ight. Consider the following two alternative, two 


outcome payoff matrix: 
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- ALTERNATIVES 


+ Ag Az 

POSSIBLE £7. Ss 7 
OUTCOMES (5) 

Ee! 9 8 


A minimex regret analysis would consider what the best. 
alternative would be for each possible outcome. Clearly 

if Ex, occurs, the best strategy is select Az. Likewise, A; 
is best, if Ex takes place. The analysis continues by 


transforming the payoff matrix into a matrix of opportunity 


losses: 
ALTERNATIVES 
: Ay Az 
POSSIBLE ie oa 0 
OUTCOMES 1 (6) 
Eo © i 


From this step, the alternative that provides the et 
non-zero opportunity loss is selected. 

This approach has certain emotional appeal, 
particularly in situations where hindsight evaluations can 
severely affect the decision maker. It may be the most 
regularly applied, at least subconsciously, of all decision 

analysis tools. A serious flaw affects this strategy. It 


can be shown that a decision maker employing the minimax 
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regret approach could be turned into a perpetual money— 
making machine {Ref. 24], violating the coherence principle. 
Additionally, the technique is subject to the problem of 
rank reversal, which negates its effectiveness in tackling 
problems that unfold in a sequential ered 

2. Stochastic Methods 

| The coherence principle implies that the decision 
maker will use all of the information at his disposal. if 
some opinion of the relative likelihood of various events is 
available, it should be explicitly brought into the > 
analysis. Information about likelihood could arise from a 
body of quantified statistical data, e.g., survey sample or 
previous experience, from which a probability distribution 
could be constructed or might reside in the field of. 
judgement and professional intuition. When dealing with the 
allocation of scarce resources to obtain some best possible 
return, it would be foolish not to incorporate as much 
relevant information into the decision making process as 
possible. Three common methods, Modal, Expected Value, and 
Expected Regret attack the decision problem in a 
probabilistic manner. All are measures of central tendency, 
and as such, tend to ignore or reduce the impact of outliers 


on the recommended decision. 
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a. Modal Analysis 
As the name implies, modal analysis seeks to 
identify the most iaKeiy occurrence, based upon some set of 
probabilities the decision maker has provided. Consider tne 
ee euaG example of alternatives and potential outcomes | 


subject to the assigned probabilities. 


ALTERNATIVES 
Ei P(BHi) 1 Aa Az Ax 
POSSIBLE Se Le es ee ae 
OUTCOMES Eii 0.2 » LO 8 £9 
Ez: 0.5 i 6 9 3 (7) 
Ba: 0.9 ; 18 5 1 


Note that the probabilities sum to one. It can be easily 
seen, that under the modal scheme, Ez is most likely, hence 
Az should be the chosen course of action. Problems with 
this method arise when the resolution between probability 
values decreases and the potential payoff matrix possesses 
large differences between some values and little Ave terence 


between others. 
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Consider the counter-example: | 


ALTERNATIVES 
, 4 
POSSIBLE SRE meee ere el ieee sds 
OUTCOMES Bis 6.21% “4 98 90 
Bal Osoo. 2. og 85 83 (8) 
Ex! 0.56 :} 18 15 15 


The modal approach would yield Ex as the preferred 
alternative, without investigating the possible effects of 
the large difference in payoff values. An argument has been 
made that modal analysis is subject to the same perpetual 
money-making criticism that afflicted the minimax regret 
method [Ref. 24]. Modal analysis can also be affected by 
the way data are aggregated. Consider the following example 
taken from Devore [Ref. 25:p. 11]. The length of service, 
in years, for 94 Supreme Court Justices who had terminated 
their services were tallied. If the length of service is 
aggregated by five year increments the following matrix is 


obtained: 





LENGTH OF SERVICE 5 NUMBER OBSERVED 
Oe 3D dd. 
2 ieee es. | 37 
16-20 28 (9) 
26° 3.95 seg 
> 36 de . 
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With this aggregation, the mode is 6 —- 15 years of service. 
However if the data are processed in ten year increments the 
mode changes.. 


LENGTH OF SERVICE NUMBER OBSERVED 


C= 2 5 
Bee 9 23 
8 - 12 14 

13-17 17 7 (10) 
18 — 22 14 
23 97 7 
26 = 32 8 
> 33 ; 6 


The mode in (10) is now 3 - 7 years of service. This 
Ti istestee that the abilities of the data to explain a 
process are sensitive to the metrics imposed by the decision 
maker. 
b. Expected Value 

Expected value calculates the average or mean 
payoff value and then chooses the alternative that provides 
the highest mean return. Each possible alternative payoff 
is multiplied by its associated probability and the results 
are summed for each alternative. Matrix (11) provides an 
example of how probability is incorporated into the payoff 


matrix to determine the best alternative. 
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ALTERNATIVES | 


POSSIBLE Ar cee See ome 
OUTCOMES Eit-0.2 3.10 8 £5 
E2! 0.5 -i 6 9 3 (11) 
Es! 0:3 53°18 5 1 . 
‘EV(Ai) } 10.4 7.6 4.8 


The expected value decision for (11) ehooses option A: as 
the alternative that provides the largest return. 

Since all the events are the result of random 
outcomes, expected value analysis requires the decision 
maker to be able to withstand short term losses or instances 
of less than ideal returns. The expected value approach may 
not be appropriate for one-of-a-kind decisions or for 
instances where exceptionally large or ruinous losses are 
associated with small, but potentially significant 
probabilities. | 

c. Expected Regret 

The expected regret method applies the. 
probabilities associated with each outcome to a regret or 
opportunity loss matrix similar to (6) above. Using (11) 
and applying the regret technique, the following payoff 


matrix is obtained: 
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ALTERNATIVES 


E | P(Ei) | A; Az Ax 
POSSIBLE eee: 
OUTCOMES Bl 0.2.6 5 7 0 
: Ee! 0.5-°% 3 0 6 (12) 
E=! 0.3 : 0 13 17 
‘'PR(Ai) | 2.5 5.3 8.1 


Again, Ai is the best alternative as it causes the least 
amount of opportunity loss. Maximizing EV(A:i) will always 
lead to the minimization of ER(Ai). As noted with expected 
value calculations, the decision maker will be subject to 
the variance associated with each of the probabilities and 
must be able to withstand short-term instances of contrary 
results. 

3. Summary 

The basic classical statistical methods for 

determining rational decisions were presented. Strengths 
and weaknesses of were noted. A limitation affecting all 
of the methods was the failure to account for individual 
preferences in regard to the magnitude of potential gains 
and losses. Methods that seeks to overcome this obstacle, 
mnie Sreauei ng rational and consistent results for | 
individual decision makers, will be presented in the 


following sections. 
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C. UNCERTAINTY AND RISK 


Uncertainty stems from the problem of dealing with 
random events where the Hike ineea of an event's occurrence 
is subject to some measure of variability. Humans have 
exceptional aifficulty dealing consistently with random 


events. Coin tosses or other binomial events where an 


objective and precise probability of occurrence can be 
specified are conceptually easier to handle than events for 
which no objective body of data exists from which to 
generate a probability distribution. Kahneman and Tversky 
(Ref. 26] have demonstrated that even individuals with 
Seat cies! training will often be subject to three 
particular biases: representativeness, anchoring, and 
availability. These biases are, "... highly economical and 
usually effective, but they lead to systematic and 
predictable errors." [Ref. 26:p. 20] Since people have 
difficulty dealing with random events, the concept of risk 
is equally befuddling. | 

The risk of an event is the probability of occurrence 
multiplied by cost. Risk is associated with notion of the 
value that may be lost or unrealized as a result of a random 
event. An additional issue is the size of the variance 
affecting the probabilities applied to the problem. Because 
values are relative, how should a decision maker obtain a 
defensible solution to the problem of hazarding current 


resources against the attainment of value at some future : 
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time when the outcome is subject to randomness? Extensive 
research has shown that individual concepts of risk change 
dramatically as the potential payoff or loss becomes very 
large even if the associated probabilities become relatively 
small. The Allais Paradox, presented below, is an example 
of this behavior. 


Suppose a person is offered the following wager: 


A: $1 Million guaranteed 
| versus 
B: $5 Million with a 10 out of a 100 chance 
$1 Million with a 89 out of 100 chance (13) 


No money with a 1 out of a 100 chance 


Allais, Raiffa, and Tversky have shown that the modal 
response is wager B [Refs. 17, 26, 27]. When the situation 
is altered so that there are large difference between payoff 
values and small differences between probabilities people do 
not follow the expected value model. Compare the following 


wager Oe I 8 


C: $5 Million with a 10 out of 100 chance 
No money with a 90 out of 100 chance 
versus (14) 
D: $1 Million with a 11 out of 100 chance 


No money with a 89 out of 100 chance 
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The expected value model selects C over D because of the 


larger expected value: 
EV(C) = $500K > EV(D) = $110K — 


The modal response however is D. People begin to focus on 
the small difference probability when the differences 
between payoffs begin to get very large [Ref. 28]. 
Objective statements that are applicable to a broad range of 
Situations are exceptionally difficult to make. The 
following sections will discuss some of the problems 
associated with the measurement criteria that have been 
suggested for decision analysis... 
1. Historical Antecedents 

Risk assessment is a very Sa practice. Records of 
assessment, valuation, and accounting procedures, dating to 
3200 B.C. have been found in the Tigris-Euphrates valley 
‘[Ref. 291. The "risk accounting" procedure that was used 
presages a modern proposal introduced to handle the risk and 
uncertainty issues surrounding nuclear energy [Ref. 30]. 
The determination of risk and subsequent recommendation of 
appropriate courses of action were placed in the hands of a 
priestly caste who used a form of double-entry bookkeeping 
to arrive at a quantified decision, carrying the weight of 


assurance and confidence. Modern antecedents, employing 
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concepts of mathematical probability and scientific analysis 
of cause and effect, effectively began with Laplace's 1792. 
aaah of smallpox deaths. 

Despite Laplace's ground-breaking effort, the 


abilities of organizations to gather relevant data and 





subsequently manipulate it in mathematically sound and 
useful fashion eeaa inca haphazard. AYenough the concept of 
insurance to spread the risk of commercial enterprises, as 
well as compensate for the vagaries of human mortality are 
ancient, consistent success in risk assessment and survival 
of risk managing entities required the establishment of 
professional actuaries. Covello and Mumpower cite findings 
of the British Government in 1867 that over 75 percent of 
life insurance endeavors failed:‘in the preceding 75 years 
[Ref. 29]. More technically sophisticated methods for 
analyzing the risk in commercial undertakings other than 
life insurance or oceanic shipping required both more 
“pigsreus codification of economic theory and more flexible 
mathematical tools. 
As Modern Perspectives 

All of the decision criteria examined so far either 
ignore the existence of random behavior, or treat the 
probabilistic likelihoods with certainty, failing to 
consider the variable inherent in random events. As 


discussed above, expected value or expected regret 
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calculations fail to capture the potential impact of the 
loss of value to the decision maker. The following example 
highlights this problem. 

Consider to wagers with identical probabilities 


of occurrence: 


Wager A: Receive $10 with probability of 0.5 
Lose $10 with probability of 0.5 


(215) 


Wager B: Receive $10,000 with probability of 0.5 
Lose $10,000 with probability of 0.5 


Evaluated from an expected value perspective, both wagers 


are identical: 
EV(A) = EV(B) = Q. 


However, if an individual participates in wager B, the 
magnitude of the loss may be radically different from that 
of wager A, depending on what proportion of net worth a 
$10,000 loss represents. It is conceivable that an 
individual could be induced to pay sum amount of money or 
part with some quantity of value to “buy out" of wager B and 
into wager A. How much money an individual would be willing 
to give up and under what conditions forms the basic 
approach to current risk analysis techniques. A significant 


obstacle to this search is the variability of risk 
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perception between heise marere.. 8s well as, the. 
variability of risk perception by individuals in different 
decision environments. Generically, endeavors subject to - 
greater risk are understood to possess either greater 
likelihood of potentially ruinous losses or instances where 
the probability of occurrence may be quite small but the 
associated event would be devastating. Crossing a busy 
highway blindfolded would be an example of the former’ while 
an accident at a nuclear power facility oe an aircraft 
landing on a house would be examples of the latter (Ref. 
31) Risk analysis attempts to provide an additional 
decision criterion that focuses on the potential variability 
of an event, as a complement to expected value analysis, 
which ignores the variance issue. There are four major 
approaches for tackling the risk problem: 

oa. Variance 

This is the usual statistical definition of 


estimated variance: 


This captures the notion that the greater the variance, the 
greater the potential risk. It does not address the problem 
of potential skewness of the distribution governing y as 


Figure 2-1 demonstrates. 
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Figure 2-1 Distributions with Same Means and Variances 
If the full probability density function is considered, 
series A would be preferred and series B considered more 
risky. 

b. Semivariance 
Semivariance attempts to concentrate on the 
perceived risky portion of the payoff distribution, 
using a pre-determined critical value: 
~*~ 
2 2 


i i 
# _@ 
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As with the variance method, distributions with equal mean 

and semivariance, but different risk potential, can be 

constructed as shown in Figure 2-2. As with the example of 
: variance as a iesetee of risk, when the full probability 
density functions of series A and B are considered, series B 


would be the riskier of the two. 


SAME MEAN AND SEMIVARIANCE 
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Figure 2-2 Distributions with Same Means and Semivariance 
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c. Critical Probability 


The critical probability approach uses 
the same format as semivariance but substitutes a 
probability value for a measure of variability. 
rs 
P(y < c) = J f (y) dy 
—o j 
Bunn cites a definition generated by Fishburn that links the 


two approaches. 


When b = 2, the semivariance method is used and when b = 0 
critical probability is operable [Ref. 17:p. 35]. The idea 
of a critical probability has certain intuitive aipant. The 
critical probability can be viewed as a goal, with reaching 
or surpassing the critical value defined as success and 
falling below the critical value defined as failure. This 
highlights the personal and transitory nature of risk 
evaluation [Ref. 26]. As with variance and semivariance, 
counter-examples are easy to create, as shown in Figure 2-3, 
_where both distributions have equal area be low zero but 


series B could be considered more risky. 
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Figure 2-3 Distributions with Same Means and Equal Area 
Below Zero 
dad. Mean-Variance Dominance 
This approach has its roots in financial 
analysis. The premise is the same as discussed for the 
nonstochastic dominance case. The outcomes of various 
alternatives are characterized by their respective means and 
variances. ‘The decision rule is to choose those options 
that possess the greatest mean and smallest variance. While 
this may lead to an "efficient" set, where dominance ‘cannot 


further winnow the choices, it may not identify a clear 
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winner. Additionally, it requires a more precise knowledge 
of the process governing the outcomes than may be available 
in some decision environments. Utility Theory offers a 
potentially coherent avenue around some of these 


limitations. 


D. UTILITY THEORY 

In 1944, John von Neumann and Oskar Morgenstern 
published their influential work, "Theory of Games and 
Economic Behavior" [Ref. 5]. Arising from a concept 
originally put forward by one of the authors in 1928, the 
principles developed in their work and embellished by 
subsequent researchers form the basis for most current 
techniques of risk assessment. Morgenstern and von Neumann 
Sought to develop a complete set of rules that would define 
"“yational behavior” and could consequently be used to guide 
actions in any circumstance. They defined rationality as 
optimal behavior in an economic sense; individuals would 
consistently seek to improve their conditions. Optimal 
behavior could be achieved through the use of "... the 
notion of mathematical expectation” [Ref. 5:p. 32]. 

1. Axiomatic Structure 

von Neumann and Morgenstern define seven key 

elements in this theory which provide a firm, logical basis 


of coherence. The theory is normative vice descriptive, 
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providing a framework that guarantees a rational result. In 
its original form it does not necessarily describe how real- 
world decision makers actually arrive at a solution. 
a. Structure 
First, the decision maker can create an order 
out of the decision situation. He can describe his 
alternatives and can create a listing of the payoffs that 
would proceed from the perfect execution of these options. 
Sophisticated models may be required to capture the 
essential elements of the decision environment, but such = 
construction is assumed to be possible. 
b. Ordering | 
Second, a preference order can be determined for 
the list of possible objectives or outcomes of a specified 
process. Given 4 choice of three items, A, B, or C, with 
the opportunity to select only one at a time, the decision 
maker can order these choices in a manner that reflects his 


desires. 
AO>BoOoC 


| This implies choice A is preferred to choice B and choice B 


is preferred to choice C. 
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c. Transitivity 
The third point is transitivity. If the 
decision maker were provided with the choices as pairs, say 
A and B, or B and C, subject to the ordering 
A > Band B>C, then it follows that A > C. 
d. Indifference 
Fourth is ina tetonence: The composition of the 
choices may be such that for some particular grouping, the 
decision maker cannot categorically state a preference 
between choices. This indifference or substitutability 
holds whether the various pairs of choices are considered 
alone or whether they are grouped into a more complicated 
option. 
e. Reduction of Compound Prospects 
Fifth, “Any compound prospect should be 
indifferent to the equivalent simple prospect with 
prohabilities computed according to the usual rules of 
probability." © [Ref. 17:p. 54]. This provides a coherent 
structure to the proposed solutions. Central to Oe nevies 
is the requirement that the decision maker has no preference 
for the time-based portion of an uncertain prospect. This 
rules out the consideration of preferences arising from the 
joy of playing the game, the particular atmosphere 
associated with a particular decision environment, ae the 


manner in which risks are presented. 
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f. Continuity 
Sixth, utility values are continuous. This 

involves the notion that for extremely noxious outcomes, a 
non-zero probability exists that would lead the decision 
maker to accept some other alternative which may be trivial 
in some other circumstance. In the normal course of events, 
an individual may be prefer a small sum of money to a zero 
payoff, say $5. Then a non-zero probability exists, such 
that, a prospect between $5 or death would be attractive. 
While such an extreme seldom occurs, it is the sort of 
problem that explicitly confronts public policy bodies when 
ruling on issues that affect the welfare of society. This 
axiom explicitly supports the construction of a utility 
curve and allows for the more precise resolution of 
utilities in regions of rapidly changing risk perception. 

g. Monotonicity | 


Finally, preferences must be monotonic. This 





implies that option A is preferred or indifferent to option 
B. if and only if, the probability of A, pa, is greater-than 
or equal to the probability of B, pe. Monotonicity provides 
a necessary and sufficient condition for the | 
preferred/indifferent ordering of various alternatives. 


Thus, the utilities of different alternatives can be 
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compared through the use of expected utility. The expected 
utility is calculated in the same manner as expected value 


leading to: 


EU(A) 2 EU(B) 


2. Limitations 

| As with any theory, there are specific limitations 
that must be observed when applying the theory | 
to a particular decision problem. 

a. Utilities Are Not Additive 
In the event that the final payoff for some 

prospect is the result of some sequence of alternatives, the 
utility of the final payoff is not equal to the sum of the 


individual payoff's utilities: 
U(A + B) # U(A) + U(B) 


This is the result of the curvature of the utility function. 
Specifically, it recognizes the possible change in risk 
perception governing alternative considerations, as the 
values of the payoffs change. The individual utilities are 
additive, only in the case of a risk-neutral decision maker 


who effectively adopts an expected value criterion. 
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b. Strength of Preferences Not Captured 
The von Neumann/Morgenstern version of 
utility theory creates an ordinal scale as opposed to on 
interval scale. The numerical ranking is imposed only for 
the purpose of sorting preferences. Consider the example of 
team rankings in college football. A team could be rated 
fifth one year and fourth the following year. This does not 
imply that the same level of progress was made, as a team 
that was ranked second one year and then ranked number one 
the next year [Ref. 32]. 
c. Not Comparable Between Decision Makers 

The evaluation of risk as an integral part of 
ety seneeey is very personal. Rapaport and Wallenstein 
have shown that, Te pe ag ES concept of risk is highly 
idiosyncratic,” and is a function of the risk-taking © 
situation or environment, personality characteristics of the 
individual(s) involved, and their training and Seer dene 
{Ref. 33]. This notion can be extended to social and 
corporate organizations, as well. A dramatic example of 
this feature of utility theory is MacArthur's decision to 
proceed with an amphibious invasion at Inchon, despite 
persistent and strident staff recommendations - the 
contrary [Ref. 34]. 

Determination of the relevant probabilities is a_ 

key to this aspect of risk assessment. As noted above, 


decision makers can be liable to the distortions caused by 
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availability, representativeness, and imprecise notions of 
causality [Ref. 26]... Additionally, decision makers often 
discount assessed probabilities once a course of action is 
decided upon, believing that they possess the ability to 
control events [Ref. SOU% 
3. Certainty Equivalents 

While the axioms of Utility Theory provide a 
template for determining whether or not an uncertain action 
is coherent, some scheme is required to help the decision 
meer order his preferences when faced with uncertain 
prospects. The use of Certainty Equivalents was developed 
for this purpose. If the payoff from a probabilistic 
outcome can be compared in some fashion with a guaranteed 
result, a quantifiable measurement of the individual's risk 
perception will emerge. 

Consider again the wager proposed in the discussion 
of modern approaches to risk assessment: 
Wager A: Receive $10 with probability of 0.5 

Lose $10 with probability of 0.5 
(15) 
Wager B: Receive $10,000 with probability of 0.5 
Lose $10,000 with probability of 0.5. 

Depending upon the characteristics of the decision maker, he 
may not be indifferent to these two wagers which would be 
classified as equal under the expected value criterion. The 


question then becomes, under what combination of 
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probabilities and payoffs would he subsequently be 
indifferent to A and B? von Neumann and Morgenstern 
formalized the notion of sereannty equivalents as a means of 
addressing this resus {Ref. 5]. The method is intuitively 
easier to grasp if the nature of the wagers is altered 
somewhat . 
a. Certainty Equivalent Assessment 
Consider the following wager: 
Wager A: $1000 with probability = 0.5 (16) 
$0 with probability = 0.5 
As the wager is presently constituted, the decision maker 


might agree to participate as there is no potential loss and 





it costs nothing to participate. Now suppose the wager is 
amended as follows: 
Wager A: $1000 with probability = 0.5 
$0 with probability = 0.5 
_ (17) 


Wager B: $300 with probability 
~=$300 with probability 


Oo 
SS) 


In this instance, the decision maker is required to choose 
between the wagers, one affected by uncertainty, and another 
one a guaranteed prospect. As presented, the decision maker 


would choose alternative (A). Based upon an expected value 
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calculation and dominance reasoning: | 


EV(A) = $500 EV(B) = $300 
and : 


EV(A) > EV(B) 


By similar dominance reasoning, if the decision maker were 
offered $600 not to participate in the wager, the maine 
would take the $600. Therefore, there is some value between 
zero and $600 which if offered to the decision maker as 
certainty, would result in the decision maker being 
indifferent to the choice of either A or B. Through an 
iterative process it should be possible to determine the 
decision maker's certainty equivalent for the wagers 
presented in (17). For argument's sake, say this value is 
¢400, i.e., if offered $400 to forego the anger between the 
uncertain prospect and the guaranteed payoff, the decision 
maker would accept the $400. This idea can be encoded into 


a payoff matrix as follows: 





ALTERNATIVES 
Et P(Bs) 1 As Az 
POSSIBLE i —— 
OUTCOMES E,: 0.5. '1000 400 
: (18) 
Es) 0.5. 4 


0 400 


At a monetary certainty level of $400, the decision maker 
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would be indifferent between A; snaon:,. Additionally, the 
sensitivity of this Certainty Equivalent (CE) to small 
perturbations could be uncovered. If the CE is changed to 
say, $410, then the decision might be to choose Az. 
Likewise, if the CE is: reduced to $390, then A: would be 
chosen. A formal presentation of this hypothesis developed 


by Bunn is: 


Preference is indifferent between 
X with probability p 
or 
Y with probability 1 - p 


and Z for certain [Refs. 17:p. 41, 5:p. 24]. 


The advantage of this principle is that, in 
theory, it provides a means of uncovering coherent choices 
in the presence of uncertainty. An additional benefit is 
that values, other than money, can be used to achieve 
coherent decision solutions. Payoff matrices more complex 
than the simple binary example presented in (18) can be 
evaluated with this approach and finer resolution of the 
curvature of the utility function can be obtained by 
successively examining segments of an initially binary 


problem. 
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b. Single Attribute Utility Functions 

The standard approach is to scale the utility 
sais belween zero snacone with the best case being 
assigned a utility of one. The resultant CE can then be 
used to further partition the utility scale for as many 
points as desired, as demonstrated by the following utility 
funclion constructed from wager (17) above. Matrix (19): 
contains the payoff-utility values and Figure 2-4 displays 


the results graphically. 


PAYOFF — UTILITY MATRIX 


ee. / U(X) 
1000 : 1.00 
FOG x 30075 
500 «:. Q.50 (19) 
37 Dues 
300 : 0.00 
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, UTILITY FUNCTION FOR (19) 
| | Payoff vs Utility 


' 
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| | 
| 
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| | 
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| 0.6 | 
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| } 
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Figure 2-4 Single Attribute Utility Function 


As can be see from the graph, intermediate utility 


values can be obtained by interpolating between points. A 





word of caution is necessary: if the decision maker feels 
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that great importance can be attached to fitted utility 
values in close proximity to each other, then a detailed 
sensitivity analysis may be required to obtain the 
resolution necessary to support the decision. In such a 
case, the decision may not be solvable graphically [Ref. 17: 
Dy LOS) « | 

The final benefit of a utility function approach 
is that the utilities can be compared using Expected Utility 
values, EU(x), thus capturing the risk assessment attributes 
of the decision-maker. This is accomplished through a 
calculation similar to the determination of expected value. 
The derived utility values are multiplied by their 
associated probabilities and then summed across the range of 
possible outcomes. A coherent solution is that prospect 
which has the highest expected utility value. 

4. Risk Attitudes and Premiums 
In wager (17) above, it was posited that the CE 

would be $400. The expected value of wager A was determined 
to be $500. Can anything about the decision maker's 
perception of risk be determined from this fact? A Risk 
Premium (RP) can be defined that is the difference between 
the CE assigned to a prospect and that prospect’s expected 


value: 


-RP(x) = EV(x) — CE(x) : 
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If the decision maker's RP is greater than zero, he. 
is considered risk-averse, i.e., he would part with some 
value equal to the RP in lieu of participating in the wager. 
If the RP is identicalty zero, he is risk-neutral. If the 
RP is less than zero, he is risk-seeking, i.e., he would be 
willing to pay some amount equal to the RP in order to 
participate in the prospect. Figure 2-5 provides a 


graphical presentation of these three cases. 


RISK BEHAVIOR 
UTILITY vs CERTAINTY EQUIVALENT 





Figure 2-5 Utility as a Function of Certainty Equivalent 








Series A, the convex curve, is ; risk-averse 
case. Series B, the straight line, is the risk-neutral 
case. Series C, the concave curve, is the risk-seeking 
case. 

It has been ated by many researchers that risk 
attitudes are not necessarily constant over the range of 
possible payoffs or utilities [Refs. 36, 37, 38]. This: 
behavior was noted above in the Allais Paradox example. 
While numerous analytic descriptions have been formulated to 
describe individual risk-perception curves PREE 2 OS To, oe 
peers approach is useful for depicting a decision 
maker's ePPeetic risk-perception attitudes. Once plotted, 
the regions in the yReiniey of inflection points can be more 
Bseeay researched e6 provide feedback to the decision maker 
about his perception of the problem under consideration. 

5. Comments on Multi-Attribute Utility Theory > 

While the von Neumann - Morgenstern formulation is 
sufficient to handle individual attributes of a problem, 
many decisions involve a combination of attributes, some of 
(hem potentealiy conflicting. The Multi-Attribute approach 
developed by Raiffa and Keeney extends the coherence of 
Utility Theory to a broader range of problems. While 
conceptually simple in formulation, in practical terms it 
requires extensive checks to ensure mutual preferential and 
utility independence for all possible pair combinations. A 


theorem provided by Keeney and Raiffa allows the set of all . 
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possible pairs involving 2” —- 2 potential tests to be 
reduced ton feats {Ref. 17]. As in the case of single 
attribute utility functions, a value for the joint utility 
function can be calculated for those instances where the 
joint utility function can be decomposed into a linear or 
multiplicative form. The multiplicative form is applicable 
for those cases where a particular attribute must be 
included in the solution (Ref. 17). Weighting factors, 
whose sum is one, are used ina manner analogous to 
probabilities in the single attribute case to provide a 
composite utility value for some particular multi-attribute 
solution set. The various solutions can een be compared 
based upon a maximization rule. 

Other approaches to the multi-attribute problem have 
been suggested, most notably Saaty's Analytical Hierarchy | 
Process (AHP) [Ref. 40]. While easier to implement in 
practice than Multi-Attribute Utility, these techniques are 
subject to criticism on a number of guounas: most notably 


coherence [Ref. 41]. 


E. SUMMARY 

This chapter has discussed the nature of the decision 
problem and the requirements that must be met to guarantee a 
rational or coherent solution. Various methods of 
decomposing the problem into simpler elements under 


stochastic and nonstochastic conditions were examined. The 
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difficulties associated with capturing potential variability 
in uncertain outcomes were presented. Utility Theory was 
presented as a means of encoding an individual decision 
maker's preferences and risk perceptions into a coherent 
solution. The concept of Certainty Equivalents was 
developed as a method for eliciting preferences and risk 
perceptions consistent with the axioms of Utility Theory. 
Some limitations of the theory were presented and the 
géneral requirements for conducting analysis of multi- 
attribute problems were defined. The existence of 
alternative means of analyzing multi-attribute problems was. 
noted along with their potential problem of providing 


coherent results. 
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III. COMMERCIAL TECHNICAL RISK ASSESSMENT 


A. BACKGROUND 

The purpose of this chapter is to examine some aspects of 
technical risk assessment as practiced by commercial 
enterprises. The Defense Science Board concluded in its 
1985 study of Department of Defense development programs 
that the creation of the IBM 360 mainframe computer, the 
Boeing 767 aircraft, the ATT telephone switch, and the 
Hughes sennencaad communications satellite all oon arco "an 
Senses an6 Size to a major weapon system aoyelo ements 
yet each took only about half as long to develop and cost 
concomitantly less." [{Ref. 2:p. 49] There are two questions 
to consider... Are the risk assessment techniques used by 
commercial enterprises successful, i.e., do they provide 
sufficient information for planning and system design? 
secondly, can risk assessment techniques used in commercial 
soeae tous be applied to Department of Defense acquisition 
operations? The first question will considered in. this 


chapter. The applicability of commercial techniques to the 


DoD environment will be discussed in Chapter IV. 





B. COMMERCIAL RISK ASSESSMENT PRACTICE 

As discussed in Chapter Tree inception of modern risk 
assessment and mathematical probability theory began with 
Laplace's study of smallpox mortality. It was only in the 
mid-nineteenth century that commercial concerns developed 
sufficient understanding of the required mathematical tools 
to begin analyzing risk in a quantitative and rational 
fashion. von Neumann and Morgenstern built upon the 
economic equilibrium theory of the Lausanne School when 
developing their Utility Theory model [Ref. 5:p..15]. Since 
the publication of their ground-breaking work in 1943, many 
additions to inventory of risk assessment and decision tools” 
have been made. Farquhar listed twenty-eight different 
techniques for tackling various aspects of the risk 
assessment problem [Ref. 39]. A natural expectation would 
be that the commercial practice of risk assessment is well 
developed and that some modicum of success is enjoyed in 
accurately identifying risky projects. Upon a closer look, 


it 1s apparent that this is not so. 


C. RISK AND AMBIGUITY 

Any commercial enterprise faces two key questions: 
(1) What do consumers want? (2) Can the firm satisfy 
particular consumer wants in a profitable manner? 
Tremendous resources are expended in the attempt to define 


consumer wants and preferences in everything from cars to 
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political parties. However, as Kosnik observed, ambiguity 
exists as to the extent of consumer needs. This complicates 
corporate efforts to satisfy these needs by matching 
existing and emerging technologies with a constantly 
changing marketplace. The marketing approach views the 
business process as a, "highly integrated effort to 
discover, create, arouse, and satisfy customer needs 
unfortunately customers cannot articulate what they need." 
[Ref. 42:p. 121] This has led to some inconsistencies in 
the application of classical Utility Theory as discussed 
below. 
1. Ambiguity Concerns 

Classical Utility Theory is based upon knowledge . 
of the underlying probability distribution governing a 
particular group of events. AS discussed in Chapter II, 
expected utilities can be calculated in these instances if 
the magnitude of potential gains and losses is Known. A 
risk premium can be specified, based upon the risk-seeking 
behavior of the firm, such that the firm would be 
indifferent to the outcome of the event under consideration. 
However, if there is ambiguity affecting the probabilities 
governing the outcomes, empirical evidence suggests that the 
associated risk premiums will be larger. Instances have 
been noted where the resultant risk premium for small, but 


ambiguous probability cases exceeded that applied to higher, 
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non-ambiguous,. loss probabilities. In these cases, firms 
seem to rely upon the "gut feelings" of the appropriate 
decision maker, than upon rigorous penemataeat analysis 
(Ref. 38). 
2. Risk Perceptions 

The size of the potential loss relative to the 
value of the company is usually the key feature in the 
practice of commercial risk assessment. This is countered 
by classical aceuenie theory. Higher performance levels, 
i.e., greater profits, will accrue to firms that undertake 
more risk [{Ref. 36]. The application of probabilities to 
estimates of potential gains and losses is the first step in 
analyzing the risk associated with a commercial undertaking... 
Those prospects possessing the largest expected value would 
be selected for exploitation. However, in some instances, 
firms may deliberately pursue prospects having a large 
expected loss because of the potential “windfall” gain that 
may accrue. In the long-run such actions would be ruinous. 
The rationale behind this behavior is poorly understood, but 
these actions can be seen by firms that are in extremis. 
In such an instance, the survival of the enterprise, as 
perceived by management, may be contingent upon the recovery 
of a "windfall" profit [Ref. 43]. This seems Peccraue for 


the application of Utility Theory to commercial decisions as 
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opposed to the strict application of Expected Value. The 
risk-seeking behavior of the firm apparently changes with 
its overall fiscal health [Ref. 38]. 
3. Problems with Utility Theory 
As stated above, the Utility Theory approach 
implies that the underlying probability distribution can be 
discovered. This can occur through elicitation of 
management's subjective ovobability estimates or through the 
careful investigation of some physical process governing the 
prospects in question. Does this, in practice, represent 
what commercial enterprises are doing? March and Shapira 
discovered that, while possessing risk preferences, 
commercial managers act ina fashion different from the von | 
Neumann and Morgenstern model. ‘Specifically, managers were 
willing to accept risks because they did not expect to bear 
them [Ref. 38]. 
a. Perceptions of Probability 

MeAggers appear to be insensitive to 
probability estimates of events. Managers focus on target 
levels and make, "sharp distinctions between taking risks 
~ and gambling." [Ref. 38:p. 1404] As discussed in Chapter 
II, the notion of a "reference gamble” to elicit utility 
values is central to the classical utility approach. 
Winkler found that if the decision maker is an "expert" in 


some field, then success will be attributed to knowledge, 
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while failure will be the fault of chance. Conversely, if 
the underlying process is Seoriy understood, then success 
will attributed to chance [Ref. 44]. 
b. The Value of Knowledge 

Given the above, the assumption could be made 
that managers possessing expertise ina particular field 
could sift through the relevant data describing some process 
and arrive at a profitable decision. This is not always the 
case. Capen, et al. conducted a study of the profitability 
of petroleum companies when bidding for Government leases. 
The problem studied by Capen, et al. was the poor success 
rate and subsequently poor profitability of firms engaged in 
Alaska oil field development. Both the Atlantic Richfield 
Company and the Humble 011 Company had cooperated in the 
“investigation of the Alaska North Slope oil fields. The 
relevant information about parcel petroleum characteristics 
was pooled and known to both companies. However, during the 
bidding process for subsequent field development, extremely 
large bid variances for the same parcels existed. The 
authors cited ratios of 100 for some parcels, with 5-10 
Beane the most common. The authors concluded that the 
winner of a particular parcel was the firm that, "most over— 
estimates the true tract value." [Ref. 45:p. 643] | 
Access to the same body of data and analysis by highly 
skilled professionals did not guarantee success. The value 


experts placed upon a particular tract was Significantly 


60 


different. Thus, recourse to expert knowledge in estimating 
the probability of success in a venture does not guarantee 


the expected pay-off. 


D. TECHNICAL RISK ASSESSMENT ISSUES 

The above quotation from the Defense Science Board (DSB) 
implied nat commercial firms may generally be successful in 
their analysis of the technical risks affecting the 
implementing of new ideas. While the projects cited by the 
DSB were resounding successes for their manufacturers, 
successes of this magnitude are not commonplace. 

1. Accuracy of Assessment 
The RAND Corporation conducted a study of 

chemical pioneer process plants. The corporations building 
the pioneer plants were all successful operations, 
possessing large capital and information resources, and 
management structures attuned to the strategic challenges 
facing their companies. The study concluded that 
satisfactory performance required in excess of ten years 
additional development work and that large cost overruns 
where common. The study attributed this to the 
establishment of fixed budget values prior to the completion 
of a final design and incomplete understanding of the 


challenges to be surmounted [Ref. 45]. 
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Similar findings sacs reported by Davis. Initial 
construction cost estimates for new facilities were 
typically less than half the eventual amount. Davis 
attributed this to under-estimating costs as opposed to true 
cost overruns. Additionally, eighty percent of new projects 
failed to achieve their predicted market share [Ref. 47]. A 
study by Battelle Memorial Laboratories discovered an 
average of 19.2 years between invention and commercial 
production [Ref. 48]. 

2. Technological Uncertainty 

A slightly different conclusion was reached ina 
recent survey of 108 San Francisco based companies spanning 
service-based to high technology firms. The ability to 
accurately ascertain the difficulties associated with new 
ventures were more acute for organizations relying upon 
advanced technologies, irrespective of company size. The 
more advanced the technology providing the company's 
profits, the greater the potentially negative consequences 
of inadequate technical risk assessment. The higher the 
level of technological advancement, the higher the level of 
opportunity costs for a successful decision. Greater 
technological complexity implies a greater degree of 
uncertainty or ambiguity for the key decision maker 


{[Ref. 49]. 
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Ghemawat agrees with this conclusion. High 
technology companies wei faded with the dilemmas of choosing 
between a high-risk, high-return prospect or, "foregoing a 
head start (and also experience related cost advantages) by 
waiting until technological uncertainty is resolved." 

[Ref. 50:p. 148] 
3. Summary 

Commercial operations have had minimal 
success in solving the probability estimation problems 
cataloged in Chapter II. Corporations are hampered by 
judgement biases which result in schedule delays and cost 
overruns. Issues of risk assessment and technological 
uncertainty are as severe for firms marketing high 
technology products as for DoD acquisition decision makers 
attempting to maintain a technologically superior force. 
Risk assessment methods that have proven to be successful 
are not suitable for firms operating in very dynamic 


markets. 


E. EVALUATION TECHNIQUES 
1. Discounted Cash Flows 
The primary method: foe evaluating. one potential of 
any project is through the use of Discounted Cash Flows 
(DCF). The basic premise is, "a dollar today is worth more 
than a dollar tomorrow." [Ref. 52] In theory, the DCF 


approach provides a rational and sound method for estimating 


63 











the cash generated by a project during different time 
periods. Projects competing for meted corporate dollars 
can be readily compared with this technique. Additionally, 
a firm can compare the projected cash flow of a project | 
against what. the firm woul earn by simply putting the money 
in the bank or through the use of similar financial 
instruments. The DCF method is not foolproof. It relies 
upon accurate and unbiased estimation of three key factors: 
Inflation, riskiness of the project over its projected life, 
and risk reduction through diversification. 
a. Inflation 

Changes an the monetary inflation rate can 
dramatically affect the profitability of a project. The 
effects of inflation become more pronounced as the time 
horizon of the projects expected life becomes longer. While 
an accurate estimate of inflation rates is crucial, the firm 
is limited in its ability to control the effects on cash 
flows. Uncertainty in this regard tends to push firms 
toward projects having quick payoffs [Ref. 51]. 

b. Project Risk 

The technical risks associated with any 

project decrease over time. This is a natural result of the 


learning process. Maintenance of a fixed risk factor for. 


the life of the project tends to under-value the cash flows 
that would occur as the project reached maturity. Older 


technology does not necessarily imply useless technology. 
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Large-scale projects can continue to provide sionitican 
cash flows long after the original payback has been 
satisfied [Refs. 51, 52]. A further complication arises 
from the methods used by firms to estimate the probabilities 
of high risk events. Firms operating in sataniisned: slowly 
growing markets have enjoyed success by employing these 
techniques. However, "for fast moving and world-wide 
industries, coe have been a disaster.” [Ref. S3:p. 124) 
c. Diversification 

A key tool available to commercial operations 
is. diversification. While there are many diversification 
techniques, the basic concept is that of portfolio 
management. The commodity future markets are, in principle. 
an example of this. An individual takes actions such that 
potential losses are balanced by gains, so that the basic 
financial position is protected. Futures contracts are 
bought or sold to match, hedge, the individual's current 
holdings and risk perceptions. The purchase of life and 
property insurance policies are other examples. 
Corporations can manage their various risk exposures through 
the use of financial instruments or through investing 
internally ins range of projects possessing varying degrees 


of risk. The idea is to compensate for risk in a way that 
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maintains the company's desired risk—-seeking position. 
Companies desiring a fou ier position would not invest in 
programs having potentially high risks [Refs. 51, 36}. 

2. Simulation Techniques : 

A modification of the basic DCF approach is to apply 

e Monte Carlo simulation to the firm's estimation of its 
business environment. Many software programs are available 
that allow the user to specify the nature of the underlying 
probability density function. A truncated log normal 
Gistribution is typically applied to new projects. Its 
shape matches the expected occurrence of budget overruns 
versus underruns: Overruns at multiples of the original 
estimate are far more Likely than very large underruns (Ref. 
52]. . While this approach allows a firm to quickly examine a 
multitude of scenarios with varying risk estimates, the 
basic problem of determining the relevance of the 
probabilities applied to the problem remains. The numbers 
produced are subject to the biases and misunderstandings 
examined in Chapter II. Strassman admits that the 
probabilities of occurrence that he uses are based upon 
personal experience and are not empirically defensible [Ref. 
52]. The simulation approach simplifies the "“number- 
crunching" aspect of analyzing a lot of prospects, but does 


not necessarily deliver an unbiased result. 
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a. Risk Adjusted Discounted Cash Flow 

Seeeesnan advocates the use of Risk Adjusted 
Discounted Cash Flow (RADCF) analysis [Ref. 52]. This 
approach splits the DCF into two segments: financial 
and operational risk. Operational risk covers all risks 
other than those associated with the cost of capital 
[Ref. 52]. Low risk projects are those possessing at least 
a 97.5 percent chance of providing profits over the life of 
the project. High risk projects are those with less than or 
equal to a 2.5 percent chance of providing profits over the 
life of the project. Moderate risks fall between these two 
extremes [Ref. 54:p. 19]. While providing better 
quantitative data for decision makers, this approach still 
begs the sueseren of how applicable probabilities should be 
determined. | 

3. The Venture Capitalist Approach 
The Venture Capitalist represents a different 

entity than the standard commercial firm. While interested — 
iri profits, he does not necessarily have to worry about | 
defending market share or finding new uses for his inventory 
of technologies. The ventaes Capitalist is generally free 
to pick his area of exploitation and arranges the timing and 


scope of his efforts accordingly. 
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a. Decision Criteria 
A Venture Capitalist typically looks at three 
items: business potential, i.e., market, pending solution of 
technical problems, expertise and commitment of the 
development team, and lastly, financial requirements (Refs. 
55, 56]. | 
b. Number of Projects | 
A vote! Venture Capitalist will look at two 
to three hundred prospects a year. Petheps ten percent of 
these will “be selected for.further consideration. Actual 
Bre eck commitments will be only two or three [Refs. 20% 
26]. The average Venture Capitalist will be involved ina 
total of five to seven projects at any time. The size of 
nS. Communion is dependent upon the Venture Capitalist's 
familiarity with the technologies am question and his 


estimate of the technical expertise of the development team. 





Financial arrangements often call for those whom the Venture 
Capitalist is backing, to invest a substantial portion of 
their net worth in the project [Refs. 55, 56]. | 

c. Expectations 

The goal of the Venture Capitalist is to 

quickly turn the project into a successful business venture 
and then divest himself of the asset. Despite the careful 
selection of projects and the relatively large commitment of 
funds and personal effort, on average, the successful 


Venture Capitalist can expect about twenty percent of his a 
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projects to be complete failures and about twenty percent to 
meet his profit goals. The remaining sixty percent will 
fall in the middle, requiring additional time to reach 
business goals and substantial divestiture effort [Refs. 55, 
56]. These results seem to match those of the larger, more 


conventional firms discussed above [Refs. 46; 48, 53]. 


F. SUMMARY 

‘This chapter examined the ways in which commercial 
| operations seek to assess their risk exposure. The use of 
classical statistical tools and Utility Theory ‘was 
discussed. Some of the common evaluation techniques were 
described, and their strengths and weaknesses noted. The 
susceptibility of commercial operations to biases in 
judgement, and over-estimation of the occurrence of 
favorable ante described in Chapter II, were highlighted: 
While quantitative approaches to technical risk are 
available, methods for probability estimation rely 
exclusively upon individual judgement and assessment. 
Standard probability estimation methods, e.g., reference 
gambles or probability wheels, are spurned by corporate 
decision makers. Risk compensation and diversification was 


also discussed. 
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IV. DoD ACQUISITION DECISION ENVIRONMENT 


A. INTRODUCTION 

The Department of Defense acquisition process is 
exceedingly complex. Managers and decision-makers struggle 
to achieve the technical performance characteristics 
required to support combatant forces within schedule and 
cost constraints. They must deal with a host of agencies 
and oversight bodies charged with monitoring all aspects of 
cost, schedule, and performance progress. Acquisition 
decision-makers must be responsive to the directives and 
requests for information from various service, Department of 
Defense, and Congressional entities that may have an 
interest in some portion of the acquisition process. [In 
1985, the Pentagon submitted almost 24,000 pages of 
documentation to Congrees. “stemming from prior years’ 
defense authorization and appropriations bills and their 
accompanying reports.” [Ref. 57:p. 76] Between 1970 and 
1985, the number of congressionally mandated reports 
increased 1000 percent [Ref. 57:p. 76]. 
| The Department of Defense acquisition process 
encompasses a multitude of functional disciplines other than 
the contracting for a particular piece of hardware or 
software. Careful consideration must be given to areas such 


as maintenance and logistics support over the life of the 
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system. The number of personnel qualifications and training 
requirements for operators and maintenance personnel must be 
taken into account. Basing requirements, support 
facilities, and environmental impact of the new system or 
its production process have to be planned. Unique test and 
evaluation facilities may be required to ensure that an 
accurate cecunce of the system's real operational 
eenponaneeras obtained. The interoperability of the 
eguipment with other services or U.S. allies must be weighed 
during the definition, design, and testing of new 
capabilities. 

An in-depth analysis of all aspects of the acquisition 
process 15 outside the scope of this thesis. This chapter 
will focus only on the assessment bf eCnniieal risk and its 
impact on the overall acquisition process as described in 
DoD Instruction 5000.2, "Defense Acquisition Management 
Policies and Procedures". [Ref. 12] This document requires 
that essential program elements 

"Include clearly defined criteria for elements leading 
to the risk assessment events. The satisfaction of 
these criteria must be documented to support the rigor 
necessary in the risk assessment process." 
{[Ref..12:p. 5-B-2] 
The intent here is to provide a broad overview of the 
decision criteria mandated by the "5000" series documents 


{Refs. 12, 13, 14] and highlight the essential chronological 


features facing the acquisition decision-maker. 
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B. ACQUISITION PROCESS OVERVIEW 
Office of Management and Budget Circular A-109 
fRef. 581, specifies requirements necessary for the conduct 
of any federal acquisition. DoD Directive 5000.1 and DoD 
Instruction 5000.2 are the governing documents for DoD 
acquisition decision-makers. They detail the procedures” 
that must be Policwea and the objectives that must be met 
for a major new system start or an upgrade to a system's 
existing capabilities. 
1. Decision Makers 

The Under Secretary of Defense for Acquisition is 
_the Defense Requisition Executive (DAE). He is supported by. 
the Defense Acquisition Board (DAB). The DAB is the top- 
level review body for Aeon weabens systems acquisition 
decisions. The DAB is chaired by the DAE. Other members of 
the Board include the Deputy Under Secretary of Defense for 
Acquisition; service Acquisition Executives of the Military 
Departments; Director of Defense Research and Engineering: 
the Assistant Secretary of Defense for Program Analysis and 
Evaluation; the Comptroller of the Department of Defense; 
and the Director of Operational Test and Evaluation. The 
Vice Chairman of the Joint Chiefs of Staff serves as Board 


Vice Chairman [Ref. 12:p. 13-A-2]. 
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a. Supporting Committees 
The DAB is supported by three committees: 

Strategic Systems (SSC); Conventional Systems (CSC); and C3I 
Systems (C3IC). The committees provide specific expertise 
to support the acquisition review process [Ref. 59:p. 3]. 

b. DAB Responsibilities : 

The DAB convenes for each milestone review to 
ensure compliance with previously specified performance 
objectives. Additionally, the DAB is to act as an 
independent assessor of program health and future viability. 
The DAB provides recommendations to the Under Secretary of 
Defense for Acquisition on the various performance, cost. 
ana schedule trade-offs that may be proposed by the new | 
system's program manager. Consideration of relevant. risk 
areas, threat, technology, design and engineering, etc., is 
an integral portion of this process during Milestone review 
(Ret. P2ep.. DSaAe25.. REt-.. +32 Dp. 4-E-1]. Specific information 
on the mechanics of the decision process are provided in the 
following section. 
2. Decision Process 

DoD uses a phased, systematic approach to analyze 
the military requirements problem, develop satisfactory 
solutions that meet performance goals, and manufacture and 
support an openaeaeanl system ina timely fashion at an 
affordable cost. Five paste Mitescones are imposed on the 


acquisition process to ensure that these requirements are 
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met. Milestones are paired with follow-on phases. The 
phases are structured toward developing an operational 
system while satisfying the interim goals set by the DAE and 
the DAB. The goal is to develop an event-driven acquisition ~ 
stvarsey, “that links program decisions to demonstrated 
aieconel vennenes in development, testing, and production." 
iRef. S59:p. 2] This structure is discussed below. 
a. Mission Need Statement 
The acquisition process begins with the 

determination of the basic system performance requirements 
necessary to either support new military obligations, combat 
emerging threat capabilities, or take advantage of new 
technologies that could substantially reduce, “ownership 
costs or ‘improve. the effectiveness of existing materiel.” 
[Ref. lz:p.-3-2] The procurement process for new capability 
only begins after a determination that military requirements 
cannot be met via changes to, “doctrine, eperdesenat 
concepts,. tactics, training, or organization.” [Ref. 12: 
p. 3-2] These requirements are formulated ina "Mission 
Need Statement” (MNS). 

| (1) Mission Need Validation. Mission needs 
identified at lower echelons are validated by a designated 
“operational” authority: a Unified Command, e.g., Atlantic 


Command, a Specified Command, e.g., Aerospace Defense 





Command, the Military Departments, the Office of the 


Secretary of Defense, or the Joint Staff [Rete P22 py 3-2). 
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If the validated requirement could potentially involve the 
use of new technologies or require a major upgrade to 
existing performance capabilities, then the requirements 
also must be validated by the Joint Requirements Oversight 
Council (JROC). 

| (2) Performance Requirements. The intent of the 
MNS is to capture perceived operational requirements ina 
broad fashion. The statement should be sufficiently 
detailed to enable subsequent technical and engineering 
studies, but must avoid identifying a specific system 
solution to the military requirements {Ref. 13:p. 2-1) 
As an example, a Mission Need could be generated for the 
collection, evaluation, and distribution of high quality 
hattlefield imagery, without specifying the use of a 
particular technique. The MNS would specify the resolution 
required, size of imagery field of view provided to 
essa bane forces, data transmission rate. cryptographic 
requirements, etc. It would then become the responsibility 
of the rerecsne DoD acquisition element to investigate and 
develop feasible technical approaches that meer the military 
requirements consistent ith schedule and cost constraints. 
DoD will often engage industry to supplement or fulfill 
these requirements. | 

(3) Threat Aesesement.. “DoD Manual 5000.2-M 

specifically requires that the mission Need Statement 


identify the specific threat to be countered and provide a 
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description of the likely threat environment [Ref. 13: 

p. 2-1-1]. This information is used at later milestones to 

validate the sufficiency of proposed technical approaches 

and help ensure consistency ee eva tual tone: * 
(4) Joint Requirements Oversight Council. The 

purpose of the MNS review by the Joint Requirements 

Oversight Council (JROC), is to ensure development of 

capabilities that may benefit more than one service or 

enhance overall force qualities in a rational manner, while 

accounting for the requirements of joint operations and 


deployment. Validation of the requirements set forth in the 





MNS is also an integral part of this review. 
b. Milestone 0: Concept Studies Approval 

Upon validation of a legitimate military need, 
the Defense nega sa Clon Executive (DAE) will direct the 
initiation of concepts Spiaiee. The purpose is to begin the 
investigation and research process necessary to define the 
Specific Lechnical characteristics or "concepts" a new 
system must possess to satisfy the MNS. The DAE will issue 
an Acquisition Decision Memorandum (ADM), specifying the 
minimum number of possible ideas that should be investigated 
and identifying the lead DoD agency responsible for 
conducting the necessary research [Ref. 59:p. 5]. At this 


juncture, no definite date is established for the discovery 
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of-a set of satisfactory solutions but a minimum set of 
requirements will be established for progression to the next 
milestone. | * 

c. Phase 0: Concept Exploration and Definition 

Successful completion of Milestone O marks the 

fornia beginning of the "Concept Exploration and Definition” 
phase. The purpose of this phase is to identify a set of 
possible technical solutions to the MNS requirements. The 
accuracy of the threat environment posed by the original MNS 
and the requirements for a new capability are also 
investigated. Additional Phase 0 objectives are to meet 
feasibility requirements specified by the DAE so that the 
program may progress to Milestone I and to begin | 
the development of an acquisition strategy [Ref 12: 
bo. 3381. | 

d. Milestone I: Concept Demonstration Approval 

Successful completion of Milestone I marks the 

official beginning of a new DoD acquisition program. The 
purpose of this checkpoint is to ensure that the 
requirements specified at Milestone 0 have been met. The 
most promising candidate solutions to the MNS are considered 
along with their availability. The accuracy of the 
projected threat environment is validated and the military 
requirements contained in the MNS examined. A complete set 
of acquisition documentation must be submitted at this 7 


review, including the Integrated Program Summary (IPS). The 
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IPS contains seven annexes addressing all relevant portions 
of the acquisition process. At this juncture, technical 
risk is specifically considered in Annex D to the IPS 

{Ref 13:p. 4-E-1). 

As mentioned in the previous section, the 
program manager is required to categorize the risks of 
threat, technology, design and engineering, etc. A ranking 
scale of low, medium, or high is used. Numericat data is 
not required. No guidance is provided as to what 
constitutes admission into one of the categories. The 
technical approach chosen will determine the magnitude of 
potential gains or losses and the likelihood of their 
occurrence. 

The ADM issued at completion of Milestone I will 
Specify a concept baseline Perce initial cost, 
schedule, and performance Sbgesaves: Exit criteria will be 
specified. These will form an interim set of goals to be 
achieved during Phase I. 

e. Phase I: Demonstration and Validation 

The purpose of Phase I is to improve the design 
characteristics and better define the expected capabilities 
of the system. Efforts are made to improve the design 
team's understanding of the technical processes involved in 
the selected approach(s). Requirements necessary for the 


successful completion of the Milestone II review are 


pursued. A Development baseline for the most promising 








alternative is created during this phase. High risk areas 
and their potential compensators are to be identified as 
well [Ref. 12:p. 3-14] 
| f_. Milestone II: Development Approval 

The key aspect of this milestone is to determine 
whether the results Be the Demonstration and Validation 
phase support continuation of the program. Specific areas 
of concern at this juncture are the validity of the 
potential threat assessment and countering Mission Need and 
whether or not the proposed technical solutions are 
understood and actually employable. Technical risk problems 
can secnd peaneiy delay passage of this milestone, resulting 
in potenlial trade-offs between original performance, cost, 
and/or schedule goals. An update IPS with revised risk 
assessment intormation, Annex D, is provided. Resource 
allocation in terms of personnel and money are reviewed. 
successful passage of this checkpoint results in a further 
refinement of base-line cost, schedule, and performance 
goals. The ADM will address specific criteria that must be 
met for passage of Milestone III, Production Approval and 
this ADM approves entry into Phase II: Engineering and 
Manufacturing Development. 


[Ref. 12:p. 3-18] 
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g. Phase II: Engineering and Manufacturing 
Development 


During this phase the most promising technical 
approach will be converted into a, "stable, producible, and 
cost effective system design.” [Ref 12:p. 3-21] The 
manufacturing process will be validated and contract 2 
compliance will be determined through system testing. The 
operational suitability of the system will be determined 
and a production baseline will be formulated. [Ref. 12:p. 
S723) 

h. Milestone III: Production Approval 

As with previous decision points, the purpose of | 
this checkpoint is to verify that the exit criteria | 
specified by the Milestone II ADM have been met. The 
results of engineering and ian Greecunadg evaluations must 
Support a conclusion that the new system design can be 
efficiently produced, is Sue ees acceptable and 
logistically supportable. By this time, most technical risk 
factors should have been eliminated. The status of 
program risk issues will be contained in the IPS. The cost, 
schedule. and performance characteristics of the new system 
will be defined by how well risk was identified and 
countered. Successful completion of a Milestone III review 
leads to the authorization for full-scale production of the 


new system. The resultant ADM will specify any program— 
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a 


specific exit criteria that must be se coaplaened during 
phase III, including retined program cost, schedule, and 
performance objectives. [Ref. 12:p. 3-24] 

i. Phase III: Production and Deployment 

The goal of Phase III is to field an operational 

capability that meets the MNS. This is achieved by a 
stable design and efficient production and support 
processes. The system performance is monitored to ensure 
compliance with MNS requirements and maintain system 
capability against projected threats. [Ref. 12:p.3-27] 


j. Milestone IV: Major Modification 
Approval 


As required. the key objective here is to 
determine whether upgrades to existing systems are required, 
prudent, and cost-effective in light of projected threat 
capabilities or emergent military requirements. technical 
risk issues re-enter at this oak. The technical approach 
| selected for a potential upgrade will affect not only the 
performance of the upgrade but also the underlying 
capabilities of the host platform. The resultant ADM will 
specify at what phase of the acquisition process the 
proposed modification will enter and approve the modified 


acquisition strategy and baseline. [Ref. 12:p. 3-29] 
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Cc. RISK ASSESSMENT PROCEDURES 

The assessment and monitoring of technical risk by DoD 
acquisition authorities te continuous requirement 
[Ref. 12:p. 6-A-3]. Senior Department of Defense managers 
Pecsgni zed In the early 1980s that broad reforms were 
required in the total acquisition mechanism. As a result, 
Deputy Secretary of Defense Frank Carlucci issued thirty. two 
Heeenetives: ah 1981. The initiatives were aimed at 
improving management control and the efficiency of the 
Scaui sition process [Ref. 57:p. 47]. Initiative il was 
directed at improving the manner in which quantitative 


program risk was identified and its magnitude estimated 





by Program Managers [Ref. 8:p. 33]. As a result. DoD 
Instruction 4245.7-M. rpansteion from Development to 
Production," was issued. | 
1. DoD Instruction 4245.7-M 

A review, conducted by the Defense Science 
Board, on acquisition program viability was conducted and 
they ee that there is, "no structural mechanism that 
can articulate with any degree of certainty the risk 
associated with the engineering and manufacturing elements 
Eo: the weapon system acquisition process." [Ref. 14:p. 1-3] 


Inadequate understanding of technical risk factors was 
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determined to be a major source of risk in itself [Ref. 14: 
p.9-8]. Currently, DODI 5000.2 PART 5, Section B addresses 
Risk Management and specifically calls out Program Manager 
use of DoD 4245.7-M to identify areas of program risk. 

DoD Instruction 4245.7-M segregates the 
acquisition process into a rational grouping of industrial 
desian problems covering design, testing, production, ~ 
facilities, logistics, and management [Ref. 14:p. 1-8]. 
Technical risk is explicitly identified and a template was 
created to aid program managers and senior decision makers 
in making better use of the technical risk information 
available to them [Ref. 14:p. 9-8]. 

_ The instruction does not specifically address 
techniques for estimating the likelihood of adverse 
consequences, nor give guidance on how decision makers 
should rank the riskiness of various alternatives. What is 
provided however, is an outline for program management. 
This outline calls for the development of a system that 
provides early identification of technical risk factors, 
instantaneous assessment of program status, and early 
indications of potential success or failure. Guidance on 
the development of such a risk assessment and monitoring 
system js absent. Program Managers are apparent ly left to 


their own initiative [Ref. 14:p. 9-9]. 
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2. The Legacy of Carlucci Initiative 11 
The commitment of senior DoD acquisition 
officials to the Carlucci Initiatives did not filter down. 
A study conducted by the General Accounting Office (GAQ) in 
1986 concluded that DoD had, "not carried through with its 
action plans on most of the Carlucci initiatives.” [Ref. 57: 
p. 48] An additional GAO study of technical risk assessment 
concluded, “The net effect of Initiative 11 on technical 
Pie Sane ssient procedures. has thus far been negligible." 
PREP. -'Gepa! 33] 
a. GAO Findings 

The GAO study cited above surveyed 
25 major. program offices from all three services. 
otandardization of assessment techniques was completely 
lacking. Has tiena a there was limited direction from the 
appropriate service acquisition authorities concerning how 
risks should be ranked. Subsequent GAO investigations of 
major programs has revealed no change in DoD technical risk 
assessment practice [Refs. 9, 10, 11]. Both DoD Instruction 
9900.2 and DoD Manual 5000.2-M [Refs. 12, 13] are devoid of 
the explicit guidance recommended by the GAO in 1986 [Ref. 
Sep Ts i 

b. Risk Compensation 

Compounding DoD technical risk problems 

is the apparent practice of examining the technical risk of 


various programs as discrete events. While some commercial 
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compensation options are unavailable to DoD management, 
e.g., hedging or other ehanean strategies, there is no 
direct mechanism for combining risk information into a 
department-wide risk profile. Risk compensation and 
reduction is tackled at the progr an level by individual 
program managers. The absence of a standardized technical 
risk quantification scheme hampers senior management's 
attempts to establish and manage a departmental risk 
portfolio. Lacking quantifiable and comparable technical 
risk data and common baseline, technology trade-off 
decisions are difficult [Ref 8:p. 51}. 


3. Corporate Information Management 
Initiative (CIM Initiative) 


Paul Strassman, Director of Defense 
Information, 15 attempting to implement CIM within the DoD. 
CIM is built upon a “business case" approach to analyze the, 
“potential costs-saving Alternatives for DoD information 
management." [Ref. S4:p. 1] A key feature of the CIM 
initiative is the use of the quantitative risk assessment 
techniques he developed in his book, The Business Value of 


Computers: An Executive's Guide [Ref. 52]. Strassman's 





goal is to conduct a Risk Adjusted Cash Flow analysis of new 
information technology acquisitions using Monte Carlo 
Simulations. The Institute for Defense Analysis created a 
spreadsheet software package that will run a user-specified 


number of trials with user-entered probability values. The 
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output of the package is a series of graphs and tables 

depicting the relative costs and benefits accruing from 

different information technology alternatives. An Expected 

Value eénparicen is the decision criterion. While the - 
program accepts user-defined probability values for the 
likelihood of achieving various outcomes, individual 
judgement is relied aeon for probability generation [Ref. 


S4:pp. 19-23]. 


D. SUMMARY 

This chapter highlighted the key features of the 
major weapon systems acquisition process. Basic decision 
criteria and. considerations. were presented. The functions 
aps the Defense Acquisition Board (DAB) and the primary 
decision authorities were listed. The importance of 
technical risk assessment to ne acquisition process was 
presented. The origin of current directives addressing 
technical risk assessment in the 1981 Carlucci Initiatives 
was noted. The continuing absence of formal and explicit 
guidance on methods to be used to assess and rank the 
technical risk associated with new system acquisitions was 
also discussed. 

The consideration of technical risk on a project-—by—- 
project basis without specific guidance for quantifying 
technical risk robs senior decision makers of valuable 


information necessary to manage a Department of Defense-wide 
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technical risk portfolio. Without quantifiable 

technical risk data, the overall impact of the risk 
associated with the technologies DoD is developing is 
uncertain. Categorization of risks as low, medium, or high 
is left to individual program managers and no baseline for 
comparison exists. As Paul Strassman has observed, "To 
understand your risks, you need to know how much money you 
could lose.” [Ref. 52:p. 217] 

While certain risk compensation strategies employed by 
commercial operations are unavailable to DoD, e.g.. hedging 
and alternative financial markets, a portfolio approach 
based upon quantifiable and comparable risk data is still 
applicable. It allows for a more rational and systematic 
improvement of any desired DoD technical risk profile. The 
everall Department of Defense technical risk position needs 
to be examined and explicitly managed to provide an overall 


technically superior military force with limited funds. 
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V. PROPOSED METHODOLOGY 


A.- BACKGROUND 

Although the axioms of Utility Theory are logically 
consistent and lead to coherent solutions in principle, 
Significant problems remain with the elicitation of specific 
utility values [Refs. 17, 26, 27]. As discussed in Chapter 
Iii, the approaches used by commercial ventures, many | 
decision makers are reluctant to use the recommended 
techniques of standard devices of 50:50 reference gambles, 
probability wheels or urns full of different colored balls. 
The most strident reclama in DoD's response to the GAO 
report on the Navy's T-45 program was reserved for use ef 
the phrase “calculated gamble" by the report's authors 
{[Ref. 9]. Probability assessment approaches are fraught 
with considerable peril and often require significant 
assistance from analysts who are outside of the decaaten 
“making organization. Significant time and resources may 
have to be committed to familiarize a decision analysis team 
with the ee of the problem. Several iterations 
involving coherence and sensitivity checks may be required 
to ensure a satisfactory result. Finally, the results of 
the analysis may require considerable distillation and 
explanation. Previous systems hailed as possible solutions 


to various risk. problems have frequently drawn criticism 
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from program managers because of these factors [Ref. 60]. 


The expertise often resides outside the program office, 
requiring the program manager to rely upon systems he may 
not fully understand and personnel who avecnet directly 
accountable to him. 
1. Limitations of Expert Opinion 
An intuitive approach to the solution of any 

eenpice problem is the elicitation of engineering or 
scientific expert opinion. While a group of experts can 
provide valuable insights into likely solution methods and 
Chea anaieidual opinions codified into a consensus, their 
seetnaeien of the associated probabilities of success or 
failure may be tremendously ambiguous and overconfident 
[Ref. 261. In a 1980 study conducted for the U.S. 
Department of Energy, Salem et al. [Ref. 31] cite an 
instance where expert opinion was elicited to determine the 
likelihood of a catastrophic seismic event involving several 
different nuclear reactor installations. The seven experts 
polled were from the fields of civil engineering, geology, 
and geophysics. 

There (sic) opinions as to the probabilities of large 

earthquakes varied by as much as four orders of 

magnitude (i.e., a factor of 10,000) in more than one 

instance. Equally as interesting is the fact that 

several of the seven participants estimated their 

uncertainties at less than a single order of 


magnitude (two estimated their general uncertainties 
as a factor of two or less). [Ref. 31:p. 34] 
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In an article on competitive bidding strategies 
for the purchase of government-owned oil reserves, it was 
noted that even though two companies had worked closely 
together on the surveying and evaluating of potential 
reserve size and had completely shared all the fede cing 
data, the subsequent bids submitted for individual parcels 
where dramatically different [Ref. 45]. 

2. Utility Assessment Issues 


Assuming the decision maker is willing to 





investigate a utility function approach to his problem, then 
his decision problem can be reduced to a number of smaller 
sub-problems. This approach is favored by various 
engineering disciplines and matches the DoD Work Break-—Down | 
Structure required during program review. As discussed in 
Chapter II, the value of utility function analysis lies in 
its ability to incorporate preferences and risk perceptions 
in a coherent fashion. | 

While risk-neutral behavior is often suggested for 
governmental operations, implying strict adherence to 
Expected Value maximization, individual decision makers and 
distinct governmental entities, i.e., DoD, Department of 
Energy, Food and Drug Administration, etc., may be extremely 
risk averse [{Refs. 17, 31] because of the immense 
‘consequences of particular decisions. Obtaining reliable 


Expected Values requires the use of probability 
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distributions characterizing a well understood process. 
This can be extremely difficult to obtain for some | | 
"one-of-a-kind" problems. 

By resorting to an analysis of his preferences 
between two separate prospects, subject to some probability 
distribution, a decision maker's risk function can be 
encoded to provide as much detail as desired. The 
assessment method involves the comparison of two prospects 


presented as follows: 


es 
i 


A guaranteed payoff 


Payoff of value G with probability p 


Payoff of value L with probability (1 - p) 


The values of G and L are not constrained to be greater than 
zero. If any three of the four sara wba? Ks Gy ae and Pp. 
are fixed, then the fourth can be determined. The decision 
maker must establish the effective end points of the 

problem, the maximum assigned a utility value of one and the 
minimum a value of zero. The assessment process then seeks 
to uncover the valuesartiaey pairs for the defined interval. 
There are four basic methods for eliciting the requisite. 
information: Certainty Equivalence, Probability Equivalence, 


Gain Equivalence, and Loss Equivalence [Ref. 17].. 
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a. Certainty Equivalence 

Recall from Chapter II that a Certainty 
Equivalent (CE) is a guaranteed payoff used to determine the 
values of. gain, loss, and associated probability of an 
opposing prospect. Gain, loss, and/or probability are 
adjusted by the decision maker so that he is indifferent 
between the value of Berean payoff and the expected value 
of a prospect operating under sanade ions of uncertainty. 

In this approach, G, L, and p are fixed 
and the decision maker is requested to provide an X, the 
value of the guaranteed payoff. Bunn states that this 
method tends Lo emphasize more risk averse behavior relative 
to potential gains and more risk-seeking behavior relative 
Lo polenlial losses [Ref. 17]. Goodwin and Wright point out 
that the manner in which the elicitation question is phrased 
substantially impacts the nature of the response [Ref. 27]. 
Goodwin and Wright use the following example of certainty 


equivalent elicitation to underscore this point: 


Insurance Formulation 
A: You have one chance out of 1,000 of losing $1,000 
B: You can buy insurance for $10 to protect against this 
loss. 
Gamble Formulation 


A: You stand one chance out of 1,000 of losing $1000 
B: You will lose $10 with certainty 
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It was discovered that 81 percent of the subjects preferred 
option B in the Insurance formulation as opposed to 56 
percent who preferred B when the prospects where framed in 
terms of a gamble [Ref. 27:p. 81]. | 

A further consideration with certainty | 
equivalence occurs if the operable probability distribution 
is skewed. If the decision problem involves only very high 
or very low peemabiiities: the 50:50 ponstruce should be 
avoided. | 

b. Probability Equivalence 

in similar fashion, probability 
equivalence fixes the values of X, G, and L and requires the 
decision maker to assess p. While this tends to provide an. 
average risk attitude, it is subject to the kinds of 
limitations discussed previously about human probability 
assessment capabilities, particularly when dealing with very 
large ai very small probability values or when attempting to 
elicit responses for small changes in likelihood, e.g., 
moving From 0.90 to 0.95 [Ref. 27]. Despite this, Certainty 
and Probability equivalence are the two more popular 
techniques. | 

c. Gain Equivalence 

In the Gain Equivalence method, value 
for certainty, loss, and probability of gain are fixed and 
the decision maker is required to assess an appropriate med 


value. An issue with this approach, as with Loss 
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Equivalence, is the affect of anchoring bias as described by 
Kahneman and Tversky [Ref. 26]. The manner of incrementing 
the Gain aus ne yet as the determination of the initial 
value, can affect the response. | 
dad. Loss Equivalence 

Loss Equivalence follows the above 
procedure with the certainty equivalent, gain, and 
probability values now fixed and the eoyfeasone ing Boceaae| 
loss value assessed by the decision maker. 

e. Summary 

As developed in Chapter II, 
theoretLically, the utility function approach allows the 
decision maker to bring his perceptions and judgements about 
the potential risk of an endeavor explicitly into a problem 
solution in a coherent manner. The four techniques 
discussed above provide methods for accomplishing this ina 
generic way but the limitations of each must be recognized. 
As pointed out in Chapter II, the development of utility 
theory arose through the consideration of monetary results. 
Much subsequent research has been spent on various means of 
equating monetary and non-monetary values [Refs. 17:pp. 
102-107, 31]. The acquisition decision maker faces a 
Similar problem. How should technology be "valued" and what 
constitutes “technical risk?" These questions will be 


considered in the next two sections. 
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B. TECHNICAL RISK 
As was pointed out in Chapter I, the force employment 
strategies favored by the United States rely upon the 
fielding and maintaining a technologically superior 
military. Many resources are expended in monitoring the 
technological capabilities of potential adversaries. As DoD 
approaches the problem of maintaining a qualitative 
technical edge, an element of uncertainty is introduced. 
Solutions to force capability problems will be selected that 
ideally will provide the requisite technical performance. 
However, the resultant system may not possess the necessary 
performance when finally fielded. Any technical performance. 
Voss then the specified value may result in a system that 
once fielded, is unable to meet ‘force employment needs in 
all intended theaters of action. Alternatively, the eyseai 
may mot possess the effective lifetime originally intended 
~hecause of failure to achieve the sodeatied technical 
performance. 
1. Technical Risk Factors 
Technical risk is affected by two related 
factors: realized technological value of the proposed system 
versus intended performance and obsolescence. 
a. Realized Technical Value 
Despite the best of intentions, a 
particular level of performance may not be achieved. given 


the characteristics of the technical approach followed. 
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This can occur because the overall level of technical 
capability was insufficient to reach the goal or because 
other program constraints forced a reduction in the 
resources available for goal attainment. As a result, the 
fielded system, even if developed within the specified time 
period may fall below its performance goals. The risk in 
this instance is anether the nearest competitor system can 
exceed the performance of the fielded system. While this 
aspect of technical risk may be mitigated by pursuit of 
parallel approaches, at some point the acquisition decision 
maner will have to commit himself to one system to meet 
force requirements. 
b. Obsolescence 

The time required to move a system from 
the design phase to IOC will ultimately affect its useful 
life. Systems are designed to provide a specific amount 
performance over a designated period. The primary 
consideration being some amount of qualitative edge, i.e., 
better than the nearest competitor by some specified amount. 
Useful system life hinges upon what else is occurring in 
related Lechnological development efforts while pursuing a 
particular approach: What is happening to the state-of—the- 
art? If the project takes longer than anticipated to bring 
to I10C, and even if technical performance goals are met, the 


useful life of the system may be significantly shortened. 
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2. Technology Valuation 

Technology has no intrinsic value. Its worth, 
like money, is only in the services it provides. Any 
attempt to measure a technological value must resort to 
Monparat ive techniques. Automobiles are valued more highly 
than a horse and carriage, by some, because of. the 
flexibility in transportation and perceived ease of 
ownership. Likewise, many capes prefer mass transit 
syslems to automobiles. The value of the automobile is 
relative to the perceptions and environment of the 
individual conducting the valuation. 

Similarly, military systems are valuable only 
in relation to potentially competitive military systems. 
The muzzle-loading rifle may have some value to antique gun. 
collectiors but its oa1ue as a present-day weapon is nil. It 
is non-competitive with its Hone of potential competitors. 
Given a mission or Sail: it should be possible to state, 
whether a particular system is capable of functioning ina 
manner sufficient to lead to the accomplishment of a mission 
or attainment of a goal. This implies the use of judgement 
by the individual or group responsible for the mission or 
goal to determine the capabilities of the technology in 
question. 

The problem then becomes of one of 
quantification versus qualification. If scarce resources 


are to be expended to obtain the services of a particular 
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piece of technology, it would be worthwhile to be able to 
state how much better one proposed technological | 
imobementation is than another. The quantification process 
forces the careful consideration of potential alternatives, 
counter-poised with the likely environments, in which the 
candidate technologies may be required to operate and the 
potential missions requiring support. The valuation method 
ae tee here is the TASCFORM™ model, developed by The 
Analytic Sciences Corporation for the Director, Net 
Assessment, Office of the Secretary of Defense, which will 


be explained in the following section. 


Cc. TECHNOLOGY VALUATION MODEL 
1. The Technology Valuation Model 
The TASCFORM™™ east. Technique for ASsessing 
Comparative Force Modernization. is means of indexing the 
technical performance characteristics of fielded military 


systems [Ref. 61]. It provides a non-dimensional number or 





figure of merit satendea to capture the multiple attributes 
associated with various weapons platforms. This approach 
provides a decision maker with an ordinal scale which can be 
used to compare and rank the technological value of 
different systems. Additionally, it allows the decision 
maker to observe how much more technical performance one 


system may have when compared with another. TASCFORM™' uses 
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an additive and multiplicative multi-attribute approach with 
“operat.ionally-oriented" subjective weighting factors to 
handle the magnitude and importance of various attributes. 
the ‘vASCFORM™ approach incorporates a variety 
of salient characteristics such as payload, range, speed, 
mobility, navigation and target acquisition measures 


{Ref. 61]. The basic form of the additive model is: 


af = kKiVi + Ke2Ve + ° . .. +k Vin . 
Where JT ts the overall platform index, Ki, . .. , Kea are 
the subjectively assigned weights and Vi, ....,. Va are the 


technology values of the respective attributes. 


The multiplicative form is: 
I = V3 (kaVa + KeVea + 2. 2. 2. tKnVe> 


This approach is used when some Vi, a particular 
technological variable, must be included in the system. 
Possible examples of this requirement would be a 
survivability, range, or payload value. 
2. TASCFORM'™ Application 

While TASCFORM™ indices have been developed for 
a wide range of forces, for the purposes of this thesis, the 
values developed for tactical and ASW aircraft were used. 


The baseline for the indices is the technology incorporated 


99 





in the F-4B, a circa 1959 aircraft. Composite technology 


values representing the synthesis of basic airframe 
performance (speed, maneuverability. range, and payload) and 
imbedded weapon system capabilities (target acquisition, 
navigation, counter-measure susceptibility, etc.) were used, 
eemiae this represents a technology index relevant to 
mission employment. 

3. Limitations 

| The indices evéaneda by the TASCFORM™ model are 
not, in themselves, predictors of the potential combat 
success or fies of the platforms under consideration. 
While not scenario specific, the weighting factors 
incorporated ~in. the Soup lev ee iece operational criteria 
relevant to a U.S.-Soviet engagement in Central Europe. 
Application of the indices to other theaters involving 
Soviet or other nationality weapons would necessitate 
readjustment of the values. The indices obtained are 
independent of the likelihood of occurrence of a particular 
scenario. Thus, while the probability of a conflict between 
U.S. forces and those of the erstwhile Soviet Union may now 
be very small, such a scenario represents an extremely 
challenging technical environment. Cost data are not 
incorporated in the model, which facilitates a dives 
approach to the assessment of technical risk unmodified by 


other considerations [Ref. 61]. 
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D. PROPOSED TECHNICAL RISK ASSESSMENT METHOD 
Any method for assessing technical risk that seeks to 
provide coherent solutions must be sensitive to both the 
axiomatic requirements of Utility. Theo and the practical 
issues of implementation. Two critical obstacles to the 
effective application of Utility Theory have been the issue 
of probability assessment and the use of reference gambles. 
The "standard device" procedure is subject to trivialization 
on the part of the decision maker. The following proposed 
method for assessing technical risk, attempts to circumvent 
these problems through the provision of an explicit 
probability distribution and a valuation system that keeps 
the technology utility function assessment process eoneinea’ 
to issues of technical value. . | 
i; Assumptions 
Following the work of Moses, Dodson, and Knight 
[Refs. 62, 63, 64], the following assumptions were made: 
a. Technological value can be quantified 
and measured. 
b. The oe of U.S. military 
Lechnological value over time can be modeled using linear 
regression techniques. 
c. A probability distribution can be 
derived from the resulting regression equation that will 


allow an acquisition decision maker to determine his 
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likelihood of obtaining a eyaten possessing some specified 
Lechnological performance value based upon the results of 
previous technological development. 

d. The performance value of the closest 


likely competitor technology can be estimated using the 


TASCFORM'™ method [Ref. 61]. 

e. Using the Certainty Equivalent technique 
described above, the acquisition decision maker can quantify 
his technological risk perceptions directly and incorporate 
these into a utility curve that would agaae him in 
determining a satisfactory approach to meeting the mission 
needs of military forces while incorporating a notion of 
technical. risk. | | 

2. System Technology Sample 

Following the work of Moses [Ref. 62], the 
statistical model used a sample of 49 U.S. Navy and Air 
Force aircraft developed between 1950 and 1979. The sample 
was restricted to tactical and anti-submarine fixed wing 
platforms for which TASCFORM™' performance index values 
where available. Source data for year of IOC and composite 
technology values were obtained from the U.S. Military 
Aircraft Cost Handbook [Ref. 65] which contained TASCFORM™™ 
composite technical values for the aircraft in the sample. 
The technology Seles contained in the Handbook were derived 
using the TASCFORM™ methodology discussed above and derived 


{[Ref. 61]. The Aircraft System Performance (ASP) values 
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were used. The ASP value adjusts basic airframe measures of 
payload, range, speed, etc., for mission requirements of 
targel acquisition, navigation, survivability, etc. and thus 
more truly reflect platform capabilities [Ref. 61]. The 
resultant values are tabulated in Table 5-1 by airframe 


designator, year of I0oc, and composite technology value. 


AIRCRAFT TECHNOLOGY MEASURES 


TABLE 5-1 , 
AIRCRAFT IOC YEAR TASCFORM COMPOSITE 
(1950-79) PERFORMANCE INDEX 
F--89C ; 20 2.46 
F=Sh/Ti oe 4.19 
F-89A 1 4.05 
F-84F 31 Ske 
F-86F ot 4.03 
F=86D od 3.68 
F-2C od Seok 
F-3A/B/C 32 9.02 
F-1B/C/M 52 SA, 
F-86H D2 5.68 
F-100A/C DZ 4.80 
F=1LA I3 » 80 
A-3A/8 Yoo 10.74 
F-102A a3 9.71 
F-6A 33 7238 
A-4A/B 33 Su93 
been! 34 ».44 
F-101A/B 04 1300 
F-100D 54 7.99 
F-8A/B/C 55 8.40 
A-1J 95 3.34 
Poor 3D 4.02 
A-1E/G/H 26 3.34 
F-104A/B 96 6.79 
F-106A/B 37 13.05 
F-105B/D 37 14.86 
A~4C 97 5.45 
F-4A/B 59 9.32 
RP og A-6A 61 13.83 
| A-4E/F 61 Tal 
F-4C/D 62 10.07 
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AIRCRAFT TECHNOLOGY MEASURES 


TABLE 5-1 
AIRCRAFT IoC YEAR TASCFORM COMPOSITE 
(1950-79) PERFORMANCE INDEX | 

P-3C 65 30233 

A-7A/B 65 12.10 

ces ee 65 18.46 . 
F-4E 66 13.96 | . 
F-111B 66 24.81 

F-4J 66 13.39 

A-7E 68 LIES 

A-7D 68 1G 217 

Fit lp 68 24.39 

5-3A 69 2052) 

F-L1iF 70 31.01 

A-4M 70 8.52 

A-6E 70 22.40 - 

F-14A ; 71 Six ok 

FioA 73 16.14 

A-10A 75 5 ee 

F-16A 78 15.69 

F/A-18A 79 . 25.42 

3. Regression Equation — 





The regression model was hypothesized to be 


linear of the form: 
FLYTECH = CONSTANT + A * YEAR + e, 


where “FLYTECH" is the TASCFORM'’’-derived composite ASP 
technology value, "YEAR" the year of platform IOC, "A" a 


regression coefficient, and "e” any residual error not 


explained by the model. The regression was run using the 
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Student Version of MINITAB, Version 1.1. Regression 
coefficients are provided below. Regression results are 


presented in Table 5-2: 
FLYTECH = -32.493 + 0.73724 * YEAR 


REGRESSION RESULTS 


TABLE 5-2 
Predictor . Coefficient Standard t-ratio p 
Deviation 
Conshant —~32.493 3. 007 -5.90° °° 0.000 
YEAR .0.73724 0.09112 8.09 0.000 
s = 5.228 R® = 58.2% R*(adj) = 57.3% 
Analysis of Variance 
SOURCE DF So MS © F p 
Regreus lon 1 1788.9 1788.9 65.46 0.000 
Error 47 1284.4 213 


LOtat 44 - 3073.3 
The regression coefficients vary slightly from those Moses 
obtained {Ref. 62] because oe the inclusion of P-3C Orion 
and S-3A Viking platforms. 
4. Regression Goodness of Fit 

The proposed regression model adequately 
fis bei the aircraft technology data. The model is 
Significant at the p = 0.000 level. Five different plots of 
the resultant regression data were constructed, aa: 
recommended by Devore [Ref. 25:pp. 498-503], to assist in 
Lhe aiaiveis of the regression's “goodness-of-fit.” Figure 
5-1 displays a plot of the regression line and the actual 


technology values over time. Figures 5-2 and 5-3 are plots 
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of Lhe vlandeardized residuals. An analysis of the data 
‘eeneaiued in these figures shows that the distribution of 
the residuals appears to be random ana most of the points 
fall with plus/minus two standard deviations of the expected 
Peatauel value of zero. Figure 5-4 displays actual versus 


estimated technology values. The graph corroborates the R=2 


value that time in the form of year of IOC is a reasonable 
explanatory variable for technical value. Figure 5-5, a 
sevinel probability plot of the standardized residuals, shows 
that the residuals follow a straight line and are contained 
within the inlerval (-2, 2). This tends to confirm the 
assumption that the error term is normally distributed. 
Figure 5-4 is a plot of estimated versus actual technology 
values. Figure 5-5 is a normal probability plot of the 
skandardized residuals. A listing of fitted values, 
residuals, standardized residuals and normal probability 


scores is contained in Appendix A. 
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TECHNOLOGY MEASURES 
PLOTTED OVER TIME 





TECHNOLOGY MEASURE 
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Figure 5-1 Plot of Technology Indices over Time 


107 





~ STANDARD RESIDUALS : 
ACTUAL vs YEAR | 








STANDARD: RESIDUAL 
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Figure 5-2 Plot of Standard Residuals vs Year 
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STANDARD RESIDUALS 
ACTUAL vs ESTIMATED 


STANDARD RESIDUALS 
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Figure 5-3 Plot of Standard Residuals vs Fitted 
Technology Indices 
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20 
FLYTECH 





Figure 5-4 Plot of Actual Versus Estimated Technology 
Values 
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NORMAL PROBABILITY PLOT 


STANDARD RESIDUALS 
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Figure 5-5 Plot of Normal Probability vs Standard 
| | Residuals 
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Following a method suggested by Montgomery 
[Ref. 66], a check was made of the standardized residuals to 
acco. ae there is a significant difference in the 
behavior of the positive residuals as compared to the 
negative residuals. An F-test with 19 and 30 degrees of 
freedom was constructed as follows: The the sum of the 
variances for the 19 positive residuals was divided by the 
sum of the 30 negative residual values. The null hypothesis 


was that this ratio should be equal to one: 


He: _s*®(it) = 1 where s*(i+) are positive residuals 
ae) and s*(i-) are negative residuals 


The results are provided in Table 5-3. 


ANALYSIS OF VARIANCE RESULTS 


TABLE 5-3 
SP (it) Se 29.4947 s*(i-) = 20.3376 
S] (i+) = 1.43994 
s“(1-) 
F = 1.43354 p = 0.184 


The test result indicates that the sums of the 
respective variances are the same at the p = 0.184 level. 
The differences in the sums of the respective variances be 
partially explained by the large effect of the F-14A and 


F-111F technology indices. These values in particular are 
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greater than two standard deviations away from the expected 
technology value for the year of IOC. 
5, Technical Advance 

Obtaining the services of a technology that 
performs better than the state of the art is an essential 
driver in the DoD acquisition process. Not only does this 
provide a potentially large quantitative edge over likely 
competitor systems at the time of IOC, it also tends to 
lengthen the use life of the system, in the presence of 
steadily progressing technological performance. 

An estimate of technology performance value 
different from the state of the art, for a given year, can 
be obtained by taking the difference between che actual and — 


estimated technology values: 
ADVANCE = FLYTECH — SOA, 


where "FLYTECH" is the TASCFORM+rm composite technology index 


and “SOA" is the state-of-the-art technology index based 
upon year of IOC estimated by the regression equation. A 
positive Advance Value indicates a system that possessed 
better than state of the art technology, while a negative 
value indicates a system that was below the SOA at time of 


IoC. The more positive the Advance Value, the greater the 
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system's qualitative edge over systems that are 
state-of-the-art. Table 5-4 provides a listing of Advance 
Values by aircraft designator. 


TECHNOLOGY VALUES: MEASURED VERSUS ESTIMATED 


TABLE 5-4 
AIRCRAFT [0C TASCFORM  $CALCULATED ADVANCE 
YEAR COMPOSITE INDEX 
(1950-79) INDEX 
F=89C 50 2.46 4.35 =1569 
F=9F/H eg 4.19 5.09 -0.90 
F-89A onl 4.05 9.09 -1.04 
F-84F ol 3 ge 5.09 0.04 
F=sorF ae 4.03 a9 L0G 
we POOD Sul 3.68 5.09 —-1.41 
Pe2e “oi 3.9! 2.09 =1.5548 
F-3A/B/C 2 9.02 ».82 o420 
Pei B se 7M 22 2229 roe =O 109 
F-86H 52 7.06 aes -0.14 
F-1OQ0A/C pe 4.80 9.82 aie 
F-11A 0 >.80 6960 “0.76 
A-3A/B oo 10.74 6.56 4.18 
F-102A . 53 O71 6.56 Si LS 
F-6A 53 7200 6.90 1.02 
A-4A/B o3 393 6.56 =2,.03 
F=-LE 24 9.44 7.30 =1.-66 
F-101A/B 54 13.39 wes 6.25 
F-100D 54 399 7 gree 18) =A Od 
F-B8A/B/C 95 8.40 8.03 0.37 
A-1iJ 39 3.34 803 -4.69 
F-9F o58) 4.02 8.03 —-4.01 
A-1E/G/H 36 3.34 8.77 =O .43 
F-104A/B 36 6.79 1 Bote 1.090 
F-106A/B 27 13:05 Gd 3.54 
F-1LOSB/D otf 14.86 9.51 3239 
A-4C 57 3.45 9192 -4.06 
F-4A/B D9 9.32 10.98 -1.66 
A-6A 61 13.83 12.46 1.3/7 
A-4E/F 61 gers 12.46 =O? 
F-4C/D 62 10.07 13519 pare eg 
P=3C » 65 30.33 15.41 14.93 
Ava? D 65 T2649 15.41 moe ee its 
F-111A 65 18.46 15.41 3.06 
F-4E 66 13.96 16.14 =2.18 
FP>iiiB 66 24.81 16.14 8.67 
P-4J 66 13239 16.14 #2679 
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TECHNOLOGY VALUES: MEASURED VERSUS ESTIMATED 


TABLE 5-4 
AIRCRAFT Loc TASCFORM CALCULATED ADVANCE 
YEAR COMPOSITE INDEX 
(1950-79) INDEX 
A-7E 68 19.77 17.62 2<«15 
A-7D 68 102.27 Le OS er es, 32) 
F-111D 68 24.39 17.62 6.77 
S-3A 69 20.21 16,35 1.86 
Polii? 70 31.01 19.09 11.92 
A-4M 70 8.52 19.09 SAA? 
A-6E 70 22.40 19.09 Seok 
F-14A 71 Slvot 19.83 ii 68 
F-15A vas: 16.14 is ieee (9) agro ets 5) 
A-10A 7D 12:12 22.78 -10,66 
F-16A 78 TD 262 24.99 29) 30 
F/A-18A 79 29.42 Zoete -0.30 


The Advance Value is calculated to provide a soiipavacive 
technology value in determining the technology utility 
function. “Tt is not the absolute measure of the index that _ 
is important, but rather the comparative difference between 
alternatives and likely competitors as viewed over time that 
provides a more meaningful measure of a system's technical 
_ advantages. An example of the value of a relatively large 
Advance Value can be seen in the TASCFORM™ F-14A 
performance value compared with "SOA" at time of I0C. 

The 11.68 1971 IOC Advance Value is greater than 
two standard devialbions above the regression estimate. The 
state-of-the-art did not achieve this value until 


approximately 1987. The attainment of this Advance Value 


provided for exceptional platform longevity. On the other 





hand, the A-6E was characterized by an Advance Value of only 
343k at the 1971 IOC, necessitating efforts in the early | 


1980s to obtain a successor system . . | 


E. TECHNICAL RISK AND UTILITY DETERMINATION 
The Advance Value information generated by the 
regression equation can now be combined with the Certainty 


Equivalent utility assessment technique, described above, to 





create a utility function that incorporates the decision 
maker's risk perception. The necessary steps are presented 
below. | 
1. Functional Limits 
Functional limits define the interval on which 
the utility function will operate. These are necessarily 
judgemental, but can be specified. The functional limits 
define the space in which the design engineers will work to 
create a system that Aearee adds taey mission needs. The 
maximum value is scaled as a utility value of one and the 
minimum acceptable value is assigned a utility value of 
aero. 
2. Utility Elicitation 
The proposed method uses the certainty 
equivalence technique, posed in the form of an "insurance" 
premium, and nests the responses working inward from the 
functional limits. Initial minimum and maximum values are 


determined from the DoD Mission Needs statement as discussed . 
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in Chapter IV. This document reflects the requirements of 
Unified and Specified Commanders, Military Departments, the 
Office of the Secretary of Defense, or the Joint Staff to 
meet national military objectives and or counter current 
threat capabilities [Ref. 12]. These values reflect the 
desired and minimum performance requirements any candidate 
system must possess to meet the requirements of the entities 
listed above. Subsequent interval endpoints are obtained 
during the utility elicitation process. 
a. Certainty Equivalence 

A certainty equivalence form, with 
probability equal to 0.5, is appropriate in this case. The 
sample Advance Value distribution appears to have a 
symmetric, normal shape [Refs. 17, 27]. This approach also 
abides by the requirement that the certainty equivalent, Z, 
lie within the interval X < Z< Y, where X is the functional 
minimum and Y the functional maximum for the interval under 
consideration. 

b. Functional Limit Determination 

Based upon enecensiyess of mission eee 
and estimated performance of likely competitor systems, the 
initial minimum acceptable Advance Value for the projected 
year of IOC can be apeeitice from the TASCFORM™ process and 
the SOA regression calculations. The functional maximum can 
be similarly determined from the regression results using 


either a plus three standard deviation value from the 
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regression estimated SOA or some increment to immediate 
pode dosed: technolody nereocmnee: Subsequent sub- 
intervals will result from the iterative utility assignment 
process. 
3. Example 

Figure 5-6 shows a representative plot of 
U.S., series A, and Soviet tactical air, series B, 
performance indices derived from the TASCFORM™” method as a 
function of time. As can be seen, U.S. platforms enjoyed 
an average advantage of about four index units when compared 
with competitor Soviet platforms. Soviet technical | 
capabilities are used for comparison purposes as these 
represent systems widely used throughout the world and pose 
a significant military challenge to U.S. forces even if not 


actually employed by Soviet forces. 


re AVERAGE PERFORMANCE INDEX 
FIXED WING TACTICAL AIRCRAFT 


AVERAGE PERFORMANCE INDEX 
1 : 


O~+~nveveaevesn 





Figure 5-6 U.S. and Soviet Technology Indices 
Over Time 
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The following is an example of how the Sinneaed 
utility assessment process would work. The utilities 
| ieotased 06 the various technology performance values are 
those of this researcher. 
a. Functional Limit Determination 
puppose the overall force structure 
requires that any new air combat system possesses a minimum 
technical pepe ormante value of at least 2.5 index units | 
above an anticipated Soviet competitor at time of IOC. The 
resultant ee ee index specifies the utility minimum 
and is assigned a value of zero. For this example, the 
minimum Advance Value will be specified as -4. The initial 
functional Advance Value maximum will be specified at 16. 
Voie value is assigned a utility value of one. 
b. Certainty Equivalence Assessment 
The initial prospect is framed in the 


following manner: 


A. Obtain a platform with an Advance Value of 16 with 
probability of 0.50 


B. Obtain a platform with Advance Value of -4 with 
probability of 0.50 


The expected value of this prospect has an Advance Value of 
6. 


C. Specify the Advance value you would accept to protect 
against obtaining a system with an Advance Value 
of -4. 
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The answer to C provides the Advance Value corresponding to 
a utility value of 0.5. the; Ewe new sub-intervals, 

(C, 16) and (-4, C), are subsequently decomposed into their 
respective utility values by means of the nesting technique 
discussed previously. Examples of possible results are 
‘presented in Table 5-5 and Figure 5-7. 

TECHNOLOGY UTILITY ASSESSMENT EXAMPLE 


TABLE 5-5 

ADVANCE U (ADVANCE) 
-4.00 0.00 
0.00 0.125 
2.00 0.25 
4.00 0.50 
8.00 0.75 
14.00 0.875 
16.00 1.00 





TECHNOLOGY UTILITY FUNCTION 


TECHNOLOGY UTILITY 
1.0 











5 10 
TECHNOLOGY ADVANCE 


_ ~*~ Series A 
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c. Analysis 
The utility curve in Figure 5-7 displays 
risk-seeking behavior on the interval (-4, 4), risk everee 
behavior over the interval (4, 10), and risk-seeking on the 
interval (10, 16). The results are consistent, in general, 
with the axioms of Utility Theory, as there is no 
requirement to display a constant risk behavior. The 
combination of risk behaviors over the sub-interval 
precludes recourse to Mean - Variance analysis. The next 
step in the analysis would be to closely exenine the 
resulis, particularly in the region of inflection points, to 
ensure that they truly reflect the decision maker's 
preferences and sensitivities. 
4, Utility Function Application 

Once the function has been determined and the 
decision maker is confident that it captures his” 
preferences, probabilities determined from the regression 
can applied. The resultant Expected Utility can then be 
used to select candidate system technologies for goal 
accomplishment. Each sandvaate would possess some Advance 
Value whose likelihood of success can be ascertained by the 
use of the normal probability distribution associated with 
the SOA regression. Given a particular Advance Value, the 
probability of achieving that value or greater can be 


directly calculated. This probability, when multiplied by 
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the corresponding utility value, gives an Expected Utility 
value (EU), which can be subsequently ranked against other 


candidates. 


F. UTILITY FUNCTION ALTERNATIVE 

Although the Utility Function approach presented above 
will capture the individual decision maker's risk 
perceptions, it joes require careful iteration. Individual 
Sana tee ties in regions of curvature changes must be 
thoroughly examined. An additional consideration is the 
organizational environment in which the acquisition decision 
is made. An allernative approach is to employ the Extended 
Pearson-Tukey (EP-T) method. 

1. The Extended Pearson-Tukey (EP-T) Method 
The EP-T method, as described by Goodwin and 

Wright, [Ret. 27] has been found useful in a variety of 
continuous distribution cases. It breaks the ranking scale 
into three segments, high, medium, and low. The three 


categories requiring decision maker estimation correspond 


to: 

a. Value which has a 95% chance of being 
exceeded (J.ow). This value is assigned a probability of 
Ope Golem 


b. Value which has a 50% chance of being 


exceeded (Medium). This is assigned a probability of 0.63. 
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c. Value which has only a 5% chance of 
being exceeded (High). This is assigned a probability of 
0.185. | | 
The probabilities reflect the symmetrical nature of the 
implied distribution and sum to one. The specific values in- 
this case can be obtained by analyzing the regression 
results in terms of the normal distribution of Advance 
Values. 

a. Example 

Using the results of the regression 
analysis, a possible risk-assessment approach using the EP-T 
method would be conducted as follows: Since the residuals 
appear to belong to a normal distribution, z-values for the _ 
respective 0.95, 0.5, and 0.05 technology values can be 
obtained from a standard normal table. The standard 
deviation value of 5.288, obtained from the regression, is 
then used to arrive at specific technology values. The 
results are summarized in Table 5-6. | 


EP—T METHOD EXAMPLE 


TABLE 5-6 
% Chance of Exceeding z-value Advance Value Risk 
95 —1.645 -8.6 ~ Low 
930 0 — 0 Medium 
OS | 1.645 8.6 High 
ae As can be seen from Table 5-4, 30 of the 49 aircraft in the 


sample would be considered low risk, 15 would rank as 
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medium risk. and four would be high risk cases. Expected 
Value (EV) methods, discussed in Chapter II, are then 
applied to Select the candidate systems which provide the 
highest EV. This may provide a faster solution to the 
problem. The disadvantage lies within the probabilities 
assigned to the three levels. As pointed out in the 
discussion of Utility Theory, Sik perception is 
intrinsically personal. While organizationally, critical 
probability levels can be assigned by fiat, these may tend 
to represent the perceptions of senior personnel, not the 
entire 


organization. 


G. SUMMARY 

This chapter began by highlighting some of the concerns 
and problems associated with obtaining the probability 
values necessary for the Utility Assessment approach. 
Recourse to experl opinion was shown to be inadequate for 
this task. The four basic methods of Utility assessment, 
Probability Rguivalence, Gain Equivalence, Loss Equivalence, 
and Certainty Equivalence, were discussed and their 
respective strengths and weaknesses noted. Two major 
factors affecting technical risk, realized technical value 
and obsolescence, were explained. A means of measuring 
technology on an ordinal scale was introduced. The - 


TASCFORM™ technology valuation model was presented and 
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discussed as a means of measuring technology for subsequent 
utility ag uesemene determination. A regression eal of . 
technology over time was constructed aaa TASCFORM™' values 
for a sample of 49 U.S. Navy and Air Force aircraft Epannng 
the period 1950-1979. The purpose of this model was 
Lwofold: To oblain a probability distribution that 
‘characterized the attainment of technology values unaffected 
by the individual biases described by Tversky and Kahnemann, 
and to obtain a measure of technological advantage, an 
“Advance vee that could be used to construct an utility 
function for varying technology values. An Utility | 
Asceasment technique, using Certainty Equivalence, was 
proposed and an example of a possible wig aseesemene 
scenario was put forward. The notion of Expected Utility 
using the probabilities determined from the regression model 
was discussed. An alternative means of risk assessment, the 
Extended Pearson-Tukey method, was discussed, and the 
example reworked using this approach. Problems associated 


with both methods were discussed. 
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Vi. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

1. Technology can be valued and measured using an | , 
ordinal scale. 

The technological characteristics of a eens system 

can be assigned numerical values in a coherent fashion. 
These values can be subsequently used to order alternative 
systems from most capable to least capable. The numerical 
differences between alternatives can provide an indication 


of how much better one alternative is than another. 


2. The increase in technological state-of-the-art over 
time can be explained by a linear regression as a first 
approximation. 

As organizations become more familiar with 
technological requirements and capabilities, the 
technological value of new systems increases over time. 
This value can be split into state-of-the-art and 
technological advance components. The projected 
state-of-the-art can be estimated via a linear regression. 


Technological advance represents capability over and above 


the state-of-the-art at the time the system is fielded. | os 








3. The probability of achieving a particular level of 
technological value can be estimated: | 

The probability distribution characterizing the 
regression residuals can be used to estimate the likelihood 
of achieving a particular technological value at a specific 
moment in time. This removes a serious problem in risk 
assessment: estimating the probability of obtaining a 
certain level of technical performance. Expert judgement 
can be focused on determining the technical requirements of 
a system and valuing the technology inherent in a new 
weapons system. 

4. Utility functions for technology can be created. 

A utility function for technology values 
characterizing a range of system alternatives can be 
created, representing the risk preference behavior of an 
individual. The utility function can be found using the 
Certainty Equivalent oaad for technology valuation systems 
possessing symmetrically distributed residuals. 

5. The Extended Pearson-Tukey (EP-T) method can be 
applied to determine the risk inherent in a range of 
alternatives. 


The EP-T method can be applied to technology 


valuation systems possessing symmetrically distributed 


residuals. This approach, while less sensitive to 





individual risk preferences, is much faster and lends itself 
to conventional Expected Value comparisons between competing 


options. 


B. RECOMMENDATIONS 


1. The Department of Defense should continue exploring a 





the utility of TASCFORM’”’-like technology valuation methods 
for technical risk assessment. | 

This would focus expert knowledge on areas of value, 
which would be more amenable to expert judgement. and avoid 
estimating probabilities by human judgement. Probabilities 
‘can be analytically determined for those cases where 4 
 ilinear regression approximates the growth of technological 
value over time. 

2. DoD should investigate the application of the 
Extended Pearson-Tukey (EP-T) method for characterizing 
technical risk Via a pilot program approach. 

While the generation of individual technology 
utili hy functions can represent an individual's risk 
perception and risk seeking behavior, a Utility Theory 
approach may be too cumbersome to employ in a complex 
organization. The EP-T method is simpler and faster to use. 
It lends itself to those applications where the probability 
distribution governing some process is symmetrical. It has 


the added advantage of supporting traditional Expected Value. ae 


analyses. 











C. AREAS FOR FURTHER RESEARCH 

a Investigate the potential dichotomies between the 
Program Manager's (PM) perception of technical risk and 
those of the Defense Acquisition Board (DAB). 

As documented in this thesis, risk perception is a 
specifically individual characteristic. The peenniead risk 
inherent in a new system acquisition project may be 
perceived differently by the DAB, the PM, and various 
supporting staffs. 

za. Investigate ordinal technology valuation schemes 
applicable for all military systems. 

A family of technology valuation schemes using — 
ordinal Sesiee would allow for ¢xplicit quantitative 
comparison. Such a device would allow senior janagenent to 
conduct technologically based eonpewisone Wetween U.S. and: 
potentially hostile forces. | 

3. Investigate methods of conducting Department of 
Defense-wide risk portfolio management. 

Methods for pooling individual quantitative project 
technical risk assessment information into a central 
portfolio should be Sepieeed. ‘Techniques for managing the 
overall DoD technical risk position should be investigated. 


This could provide senior decision makers with a more useful 
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risk indicator of DoD's exposure to technical risk. It 
would also provide quantitative data on which to base 


coherent departmental risk seeking or risk avoiding actions. 
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APPENDIX A 
TECHNOLOGY REGRESSION DATA 


- AIRCRAFT IOC YEAR TASCFORM CALCULATED 

3 : COMPOSITE  STATE-OF-THE 
INDEX ART 
F-89C . 50 2.46 4.35 
F-9F/H 51 4.19 5.09 
“ F-89A 51 4.05 5.09 
F-84F 51 5.13 5.09 
F-86F 51 4.03 5.09 
F-86D - 51 3.68 5.09 
hoc 541 3.91 5.09 
F-3A/B/C 52 9.02 5.82 
F-1B/C/M 52 5.29 5.82 
F-86H . 52 5.68 5.82 
F-100A/C 52 4.80 5.82 
F-11A 53 5.80 6.56 
A-3A/B 3 10.74 6.56 
F-102A 53 671 6.56 
F~6A 53 7.58 6.56 
A-4A/B 53 3.93 6.56 
F-1E 54 5.44 7.30 
F-101A/B 54 13.55 7.30 
F-100D 54 5.99 7.30 
F-8A/B/C 55 8.40 8.03 
A-1J 55 3.34 8.03 
P-9F | 55 4.02 8.03 
A-1E/G/H 56 3.34 8.77 
F-104A/B 56 6.79 8.77 
F-106A/B 57 13.05 9.51 
F-105B/D 57 14.86 9.51 
A-4C 5 5.45 9.51 
F-4A/B 59 9.32 10.98 
A-6A 61 13.83 12.46 
A-4E/F 61 7227 12.46 
F-4C/D 52 10.07 13.19 
P~3C 65 | 30.33 15.41 
A-7A/B 65 | 42-10 15.41 
F-111A 65 18.46 15.41 
F—4E 66 13.96. 16.14 
F~111B 66 24.81 16.14 
F-4J 66 13.39 16.14 
A-7E 68 19.77 17.62 
A-7D. 68 16.17 17.62 
F-111D 68 24.39 17.62 
n  -§-3A 69 20.21 18.35 
F-111F 70 31.01 19.09 
A-4M 70 8.52 19.09 
A-6E 70 22.40 19.09 
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APPENDIX A 
7 | TECHNOLOGY REGRESSION DATA | 
- AIRCRAFT. ToC YEAR TASCFORM CALCULATED 


‘COMPOSITE STATE-OF-THE 

INDEX ART 
F-14A 71 31.51 19.83 7 
F~15A 73 16.14 21.30 
A-10A 75 $2213 22.798 ee 
F-16A 78 15.69 24.99 ’ 


F/A-18A 79 20644 2076 
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- APPENDIX A 
TECHNOLOGY REGRESSION. DATA 


AIRCRAFT IOC YEAR ADVANCE STANDARD NORMAL 
| RESIDUALS PROBABILITY 
VALUES 
F-89C 7 50 -~1.89 -~0.91478 ~1.10394 
F-9F/H 51 -~0.90 ~1 05636 -1.45013 
. F-89A 51 aT 04 0.80981 1.01428 
: F-84F 51 0.04 —0.78958 ~1,01428 
-F-86F 51 ~1.06 ~2.08095 ~1.84145 
F-86D 51 Aiea. -~0.51618 ~0.65252 
F-2C 51 -1:18 ~1.00697 ~1.20356 — 
F-3A/B/C 52 3.20 0.26118 0.47355 
F-1B/C/M 52 -0.53 0.64542 0.85598 
F-86H 52°. ~0.14 —~0.28700 —~0.30897 
F-100A/C 52 402 0.41603 0.59076 
Paida 53 -~0.76 —0.64585 -0.85598 
A-3A/B 53 4.18 ~2.14163 —~2, 23702 
F-102A 53 3.15 ~0.10807 0.15261 
F--6A 53 1.02 -~0.36498 _ -~0.41745 
A-4A/B 53 ~2.63 ~0.23414 -~0.15264 
F-1E 54 ~1.86 0.61989 0.78453 
F-101A/B 54 6.25 -~0.42871 -~0.59076 
F-100D 54 <4 31 ~0.53953 ~0.71693 
F-8A/B/C 55 0.37 ~0.32561 ~0. 36267 
A-1d 55 -4.69 -~0 60852 -~0.78453 
F-9F 55 a4 701 0.19452 0.41745 
A-1E/G/H 56 -5.43 0.06684 0.36267 
F-104A/B 56 -1.98 -0.78286 ~0.93217 
F-106A/B 57 3.54 ~0.17934 0.05069 
F-105B/D 57 5.35 ~0.15207 0.10151 
A-4C 57 -~4.06 2.29787 1.61546 
F~4A/B 59 -1.66 ~1.03021 —1.31686 
A-6A 61 1.37 ~1.90119 ~1.61546 
A-4E/F 61 -~5.19 -0.06761 0.20411 
F-4C/D 62 ac pe 0.00460 0.30897 
P-3C 65 14.93 ~0.27914 ~0.25618 
A-7A/B 65 -3.31 -~0.21065 —0.10151 
F-111A 65 -3.06 —-0.03196 0.25618 
F-4E 66 -2.18 -~0.37473 -0.47355 
F-111B 66 8.67 —0.20674 ~0.05069 
F-4J | 66 ~2.75 ~0.25810 ~0.20411 
A-7E 68 215 ~0.20370 0.00000 
A-7D 68 ~1.45 1.17210 1.20356 
F-111D 68 6:77 0.60925 0.71693 
- §-3A } 69 1.86 -0.38799 —0.53122 
“ F-111F FO. + 11.92 1.03150 1.10394 
A-4M 70 ~10.57 0.68122 0.93217 
A-6E 70 3.31 0.58843 0.65252 
F-14A 71 11.68 1.68065 1.45013 
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APPENDIX A 
| TECHNOLOGY REGRESSION DATA 
AIRCRAFT IOC YEAR ADVANCE STANDARD NORMAL 
RESIDUALS PROBABILITY 
VALUES 
Pelion £ 72 art prae e 1331825 £31686 - 
A-10A 72 =10.66 2.33663 1.84145 
F-16A , 78 =9.30 2.89203 2.20702 Aa 
F/A-18A 79° =0:-30 0.35896 Oc OIL Ze * 
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